Sonnet 5 review: I ran 64 generations to find out if it's worth it

What It Covers

Host introduces the "How I AI Bench," a repeatable evaluation framework testing Claude Sonnet 5 against GPT-5.5, Gemini 2.5 Pro, Opus 4.8, and Sonnet 4.6 across 64 generations spanning PRD writing, UI prototyping, agentic coding, and voice personality tasks, revealing surprising model rankings.

Key Questions Answered

• •Benchmark design: Build repeatable AI evals using frozen inputs, blind scoring, and a structured rubric rather than one-off vibe checks. Claude Code can scan past session history stored on your desktop to suggest relevant benchmark tasks tailored to your actual workflows, making setup faster and more personalized than starting from scratch.
• •Sonnet 5 pricing window: Sonnet 5 launches at $2 per million input tokens and $10 per million output tokens, with Anthropic confirming prices rise after summer 2025. For teams running high-volume agentic workloads, testing and locking in usage now captures near-Opus performance at a significant cost discount before the pricing structure changes.
• •Model-by-task routing: No single frontier model wins across all tasks. GPT-5.5 produces the most comprehensive PRDs, Sonnet 4.6 performs best for UI prototyping and conversational agents, and Opus 4.8 handles dense, complex UI generation. Routing prompts to the right model by task type outperforms defaulting to one model for everything.
• •Human vs. LLM judgment gap: When the host's 70% human-weighted scores were combined with 30% automated LLM scores, rankings flipped significantly from pure LLM evaluation. LLM judges cluster scores near the middle of the scale and miss visual taste signals, making human review essential for design and writing quality assessments.
• •Agentic benchmark saturation: Standard multi-step coding tasks no longer differentiate frontier models because GPT-5.5, Gemini 2.5 Pro, Opus 4.8, and Sonnet 5 all score similarly. Effective agentic evals need harder, more specialized tasks. Retiring saturated benchmarks and replacing them with higher-difficulty challenges is necessary to surface meaningful capability differences between models.