Managing AI Coding Costs at Scale

What It Covers

Databricks co-founder Patrick Wendell explains how the company manages AI coding costs across thousands of engineers, covering model routing strategies, gateway architecture, security controls for autonomous agents, and why the next major challenge is building a company-wide knowledge layer that gives AI tools the organizational context they need to operate effectively.

Key Questions Answered

• •Model Price Deflation as Primary Lever: The single largest cost reduction mechanism is not restricting usage but positioning to capture continuous model price drops. When a new model matches previous performance at lower cost, the savings on high token volumes are substantial. Databricks runs weekly evaluations against their own codebase to identify which models sit on the current quality-cost Pareto frontier, typically maintaining a working set of three to five models across vendors.
• •Gateway-Centralized Control Plane: Route all inference traffic through a single internal gateway regardless of which client tool engineers use. This architecture enables cost visibility, dynamic model routing, zero-retention enforcement with foundation model providers, and anomaly detection — without restricting tool choice. Databricks productized this as Unity AI Gateway after building it internally, but any comparable LLM gateway achieves the same narrow-waist control point across a large engineering organization.
• •Dynamic Routing Over Manual Model Selection: Engineers default to the most capable model when left to choose, so automate routing decisions instead. Build classifier models that evaluate each request and assign it to the cheapest model likely to produce acceptable output. Cache economics matter significantly here — cache reads cost roughly 10x less than non-cache reads — so routing logic must account for when switching models destroys cache value and whether the cost savings justify rebuilding it.
• •Behavioral Risk Scoring for Agents: Static OAuth scopes are insufficient for controlling autonomous agents. Model agent behavior dynamically, similar to credit card fraud detection: score each action for risk, automatically terminate runaway sessions, and alert the deploying engineer when anomalous patterns emerge — such as an agent scanning an entire Google Drive or copying data to a public endpoint. Pair this with network sandboxing in cloud development environments to constrain where agents can reach.
• •Offline Evaluation Benchmarks Before Deployment: Avoid discovering model quality regressions through engineer complaints after a full rollout. Build an evaluation suite using a point-in-time snapshot of your own codebase, then run it within one hour of any new model release. Databricks found that some model versions scored worse on both cost and quality simultaneously, allowing them to skip those versions entirely rather than waste weeks on a degraded developer experience across thousands of engineers.
• •Context Layer as the Next Bottleneck: Model reasoning quality is no longer the primary constraint on AI productivity at scale. The binding limitation is organizational context — internal terminology like pricing units, conflicting metric definitions across teams, and knowledge distributed across Slack, data warehouses, and documents with inconsistent access controls. Building a searchable, access-controlled knowledge layer that incorporates recency and source authority signals — analogous to how Google replaced Yahoo's manually curated directory — is the next infrastructure problem to solve.