How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal
Episode
40 min
Read time
2 min
Topics
Investing, Fundraising & VC, Leadership
AI-Generated Summary
Key Takeaways
- ✓Agent-driven benchmarking: Instead of running a handful of benchmarks manually, use a coding agent to exhaustively test every open-source solution across a matrix of options. Ankur ran week-long continuous experiments comparing column store formats and execution engines, discovering bloom filters outperformed alternatives — a conclusion no human team would have resourced properly.
- ✓The "agent line" framework: Evaluate every meeting, decision, or direction-giving interaction by asking whether an agent with the same information could produce the same outcome. Ankur argues this line keeps rising, and teams that map their workflows against it consistently free up significant maker-schedule time for deep technical work.
- ✓Evals as PRDs: Treat evals as the modern product requirements document — written in prose describing success criteria, supplemented with concrete examples, and encoded so outcomes can be quantified. This shifts AI product development from defining implementation details to defining what success looks like, then letting models figure out the how.
- ✓Human taste as a calibration loop: Braintrust runs quantitative evals continuously, then brings in designer David for periodic vibe checks — roughly every few days. When David identifies failures, those observations get encoded back into scoring criteria. This scales one person's taste across more outputs without replacing their judgment or reducing their value.
- ✓CI as the velocity multiplier: When AI-assisted teams feel shipping velocity slow down, the correct response is pausing to improve CI rather than pushing more code. Ankur frames every engineer as a platform builder whose primary job on AI products is constructing a feedback pipeline — from real-world data to evals — not prompt engineering or framework selection.
What It Covers
Ankur Goyal, CEO of Braintrust, explains how engineering teams can use coding agents to run exhaustive technical benchmarks, how evals function as modern PRDs for AI products, and why CI investment is the highest-leverage move for teams shipping AI software faster without sacrificing quality.
Key Questions Answered
- •Agent-driven benchmarking: Instead of running a handful of benchmarks manually, use a coding agent to exhaustively test every open-source solution across a matrix of options. Ankur ran week-long continuous experiments comparing column store formats and execution engines, discovering bloom filters outperformed alternatives — a conclusion no human team would have resourced properly.
- •The "agent line" framework: Evaluate every meeting, decision, or direction-giving interaction by asking whether an agent with the same information could produce the same outcome. Ankur argues this line keeps rising, and teams that map their workflows against it consistently free up significant maker-schedule time for deep technical work.
- •Evals as PRDs: Treat evals as the modern product requirements document — written in prose describing success criteria, supplemented with concrete examples, and encoded so outcomes can be quantified. This shifts AI product development from defining implementation details to defining what success looks like, then letting models figure out the how.
- •Human taste as a calibration loop: Braintrust runs quantitative evals continuously, then brings in designer David for periodic vibe checks — roughly every few days. When David identifies failures, those observations get encoded back into scoring criteria. This scales one person's taste across more outputs without replacing their judgment or reducing their value.
- •CI as the velocity multiplier: When AI-assisted teams feel shipping velocity slow down, the correct response is pausing to improve CI rather than pushing more code. Ankur frames every engineer as a platform builder whose primary job on AI products is constructing a feedback pipeline — from real-world data to evals — not prompt engineering or framework selection.
Notable Moment
Ankur described hand-writing an eval script from scratch over a weekend — no autocomplete, no Copilot — after a vibe-coded version ballooned to 3,000 lines and stalled. His rule: when agents fail, close the session, improve the eval criteria, and restart from zero.
Episode Transcript
And still in as I say, the year of our cloud 2026, I still talk to engineers that say AI on our most complicated things cannot do a good job. I so viscerally disagree with. There's no staff engineer who is running as many rigorous benchmarks and trying out different algorithms and analyzing ideas manually than someone who's using an agent. Everyone should take a hard look in the mirror and reevaluate how they spend their time. There's a lot of interactions that you have or direction that you're giving or decisions that you're making, and I think, like, many of these things to me fit below the agent line. I think the agent line keeps going up. Why do you think this concept is so important to understand? And how can you just demystify it for folks who are a little intimidated Now that models are so good at actually writing code, one of the best things that we can do is create really hard evals. And if you create the right tests and success criteria for a model, then it can be really creative, and it can work on this stuff in the background and actually try to improve a bunch of things. I have a lot of people saying, wow. If I go as so far as to turn my own taste or my own skills or my own expertise into a system, I'm functionally just building my own replacement. We're able to have David's palette applied to more things. I think the quality bar that we're able to hit is higher because we're able to get more things to that bar. Welcome back to How I AI. I'm Claravo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, I have Anur Goyal, the CEO of Braintrust, and this is a technical one. So if you're a senior or staff engineer or a VP of engineering or a CTO, it's when you're really gonna wanna pay attention to. And we're gonna talk about how coding agents can help you bite off really technical architecture and infrastructure work in a way that no other human engineer could before. We're also gonna demystify evals for folks and just show you exactly how you can use them to make your AI products better without having to touch a thing. Let's get to it. This episode is brought to you by Guru, the AI layer layer of truth for your company's knowledge. Here's the problem. Your AI is only as good as the information you feed it. Most companies are getting confident but wrong answers from AI because their underlying knowledge is outdated, incomplete, or just plain incorrect. Bad information doesn't just slow you down. It costs you money and puts you at risk. Guru solves this by adding a verification layer between your company's knowledge and AI tools. Instead of just hoping your AI gets it right, Guru automatically scores …
Get the full transcript (7,671 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 37-minute episode.
Get How I AI summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from How I AI
Build your own company brain: the enterprise AI playbook from Stripe’s engineering team | Sharadh Krishnamurthy
Sep 7 · 50 min
Software Engineering Daily
AURA and Open-Source Agents for Production Operations
Jul 14
More from How I AI
GPT-6 Astra is a banger - here’s everything I’ve built
Sep 3 · 32 min
Machine Learning Street Talk
The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
Jul 1
More from How I AI
We summarize every new episode. Want them in your inbox?
Build your own company brain: the enterprise AI playbook from Stripe’s engineering team | Sharadh Krishnamurthy
GPT-6 Astra is a banger - here’s everything I’ve built
Grok Bot vs. OpenClaw: How I replaced my entire agent stack
How I turned Claude into a self-improving PM assistant | Daniel Blum (PM, Melio)
I spent $20,000 on Devin in a month. Here’s what I learned | Ryan Carson (solo founder)
Similar Episodes
Related episodes from other podcasts
Software Engineering Daily
Jul 14
AURA and Open-Source Agents for Production Operations
Machine Learning Street Talk
Jul 1
The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
In Good Company with Nicolai Tangen
Jun 17
Snowflake CEO: Scaling Data, AI Agents and the New Software Era
Venture Stories
Mar 5
Parth Patil on Coding Agents, Building Reid AI, and What It Takes to Operate at the Frontier
Product School Podcast
Mar 4
Zapier VP of Product on Orchestrating 800+ AI Agents to Manage Everything | Chris Geoghegan | E286
Explore Related Topics
This podcast is featured in Best AI Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into How I AI.
Every Monday, we deliver AI summaries of the latest episodes from How I AI and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime