Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future | Dianne Penn

What It Covers

Dianne Penn, Anthropic's first technical PM, traces the company's growth from five product engineers in 2023 to a frontier AI lab shipping multiple model series per quarter. She covers how product management is evolving around evals, token experimentation, and agentic systems, using Claude Code, MCP, and Claude Design as concrete examples of labs-driven product development.

Key Questions Answered

• •Evals Replace PRDs: Anthropic's research PM team uses evaluation sets as the primary artifact for defining product work, replacing traditional product requirement documents. When users reported Claude hallucinating, the team dug into transcripts to identify whether tool calls failed or knowledge synthesis broke down, then generated 30–40 reproducible failure examples as a structured eval. This eval runs against every new model version to measure improvement, making user pain points measurable and actionable for researchers rather than vague complaints.
• •Token Experimentation as Competitive Advantage: Spending heavily on token usage now replicates how knowledge workers will operate by 2028, when compute costs drop significantly. Penn reframes this not as raw token spend but as structured experimentation frequency. Anthropic's most creative internal thinkers spend extensive time with every new research model version, using hands-on usage to generate product ideas that cannot emerge from strategy documents alone. There is no substitute for direct model interaction when the technology moves this quickly.
• •Model-Product Flywheel: Claude Opus 4.5's breakthrough moment required both a frontier model and a frontier product simultaneously. Claude Code accelerated adoption of Opus 4.5, while Opus 4.5 unlocked Claude Code's full potential. Neither would have achieved the same impact independently. This bidirectional dependency means PM teams must build product surfaces capable of showcasing model capabilities before those capabilities fully arrive, requiring forward-compatible product architecture planned one to two model generations ahead.
• •Emergent Capabilities Are Discontinuous: Scaling law papers show model loss decreasing smoothly with compute, but capability graphs show sudden discontinuous jumps — models go from being unable to calculate one plus one to doing it reliably at a specific training threshold. These jumps are not precisely predictable in advance, which is why evals and safety red-teaming must be in place before training completes. Without systematic testing infrastructure, significant new capabilities can emerge undetected inside a deployed model.
• •Labs Structure for Zero-to-One Bets: Anthropic's Labs team operates with small pods, sometimes starting with a single engineer, pursuing discontinuous bets outside the core roadmap. The operating principle is strong conviction about a theme combined with loose attachment to the specific prototype. Bets that fail get shelved and revisited one to two model generations later rather than abandoned permanently. This structure produced Claude Code, MCP, Claude Design, computer use, and tool use — most of Anthropic's highest-impact product launches.
• •Hands-On Building Is Non-Negotiable for Managers: Senior PMs and product leaders at Anthropic follow identical onboarding plans to early-career hires, including reading consented user transcripts, talking to customers, and personally owning one to two workstreams during each model release cycle. Penn deliberately carves out time to ship directly, not just manage, in order to maintain accurate intuition about how quickly models are improving. Leaders who only receive secondhand reports cannot reliably evaluate what good AI product experiences look like.
• •Claude's Pushback Behavior Improves Output Quality: Anthropic's alignment and safety work trains Claude to disagree with users at appropriate moments rather than defaulting to agreement. This characteristic, counterintuitively, makes Claude more useful as a thinking partner. Penn uses Claude to pressure-test decisions like model pricing strategy, specifically because it surfaces objections rather than validating existing assumptions. A model that only confirms user beliefs raises the quality of individual thinking less than one that introduces friction and alternative framings.