How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal

What It Covers

Ankur Goyal, CEO of Braintrust, explains how engineering teams can use coding agents to run exhaustive technical benchmarks, how evals function as modern PRDs for AI products, and why CI investment is the highest-leverage move for teams shipping AI software faster without sacrificing quality.

Key Questions Answered

• •Agent-driven benchmarking: Instead of running a handful of benchmarks manually, use a coding agent to exhaustively test every open-source solution across a matrix of options. Ankur ran week-long continuous experiments comparing column store formats and execution engines, discovering bloom filters outperformed alternatives — a conclusion no human team would have resourced properly.
• •The "agent line" framework: Evaluate every meeting, decision, or direction-giving interaction by asking whether an agent with the same information could produce the same outcome. Ankur argues this line keeps rising, and teams that map their workflows against it consistently free up significant maker-schedule time for deep technical work.
• •Evals as PRDs: Treat evals as the modern product requirements document — written in prose describing success criteria, supplemented with concrete examples, and encoded so outcomes can be quantified. This shifts AI product development from defining implementation details to defining what success looks like, then letting models figure out the how.
• •Human taste as a calibration loop: Braintrust runs quantitative evals continuously, then brings in designer David for periodic vibe checks — roughly every few days. When David identifies failures, those observations get encoded back into scoring criteria. This scales one person's taste across more outputs without replacing their judgment or reducing their value.
• •CI as the velocity multiplier: When AI-assisted teams feel shipping velocity slow down, the correct response is pausing to improve CI rather than pushing more code. Ankur frames every engineer as a platform builder whose primary job on AI products is constructing a feedback pipeline — from real-world data to evals — not prompt engineering or framework selection.