How Companies Are Becoming AI Token Efficient

What It Covers

As AI agent adoption drives token consumption to unsustainable levels, companies like Walmart and Uber are imposing spending caps while a new category of token efficiency tools emerges. The episode examines architectural strategies, model routing systems, and benchmarking shifts that define competitive AI deployment in 2025.

Key Questions Answered

• •Token cost reality: Per-token pricing is a misleading metric for enterprise AI budgets. The actual cost is tokens multiplied by price multiplied by correction attempts. A cheaper-per-token model that "overthinks" tasks routinely costs more per completed outcome than a pricier, more concise model — a dynamic researchers call the "overthinking tax."
• •Efficiency benchmarking: Artificial Analysis now tracks a two-axis quadrant chart plotting intelligence index score against output tokens consumed. Claude Opus 4.8 scores slightly above GPT-5.5 but burns 80–90% more tokens to achieve it, placing it outside the most attractive quadrant despite leading on raw capability scores alone.
• •Model routing over brute force: Harvey AI and Fireworks AI demonstrated that routing tasks selectively — using GLM 5.1 as the primary worker and invoking Claude Opus only 0.83 times per task on average — beat Opus on both quality and cost. Post-training Kimi K2.6 achieved frontier-level legal performance at 11 times lower cost than Opus alone.
• •Four architectural levers for token efficiency: Glean CEO Arvind Jain identifies context quality, model routing, continual learning, and harness design as the primary variables controlling token spend. Systems that document prior successful executions avoid re-paying exploratory reasoning costs repeatedly, reducing redundant token consumption on repeated enterprise workflows.
• •Productized routing infrastructure: Factory Router automatically selects the optimal model per task, delivering equivalent performance to Claude Opus 4.7 at 20–25% lower cost. Perplexity's hybrid agentic inference splits agentic workflows between local hardware and cloud servers, automatically routing sensitive data locally while sending compute-heavy tasks to cloud inference.