Optimizing Agent Behavior in Production with Gideon Mendels
Episode
52 min
Read time
2 min
Topics
Productivity, Investing, Startups
AI-Generated Summary
Key Takeaways
- ✓Evaluation Dataset Bootstrap Strategy: Build evaluation datasets by capturing production failures and user complaints rather than creating synthetic data upfront. When users report incorrect agent responses, document the input, expected output, and context as test cases. Starting with just 20 real-world samples provides enough foundation to run optimization algorithms and prevent regressions, making evals practical rather than theoretical.
- ✓Prompt Optimization as Search Problem: Treat system prompts, tool descriptions, and configurations as hyperparameters in a search space. Algorithms like JEPPA use LLMs to analyze failed test cases, suggest new prompt candidates, and iteratively improve performance. LangChain's JSON schema prompt improved from 12% to 96% accuracy in two iterations for under one dollar in API costs, demonstrating rapid, cost-effective optimization.
- ✓Configuration Management Over Version Control: Store prompts and agent configurations in a centralized registry rather than embedding them in code repositories. Applications fetch current configurations at runtime, enabling product managers to update prompts without redeployment, support AB testing across traffic percentages, and implement canary deployments. This separates agent behavior updates from application deployment cycles.
- ✓End-to-End Testing Priority: Start evaluation efforts with system-level tests that validate complete agent workflows before building unit tests for individual components. The easiest high-value evaluation checks whether agents call the correct tools given specific contexts, essentially treating tool selection as a classification problem. This provides broad coverage without requiring detailed graph traversal validation.
- ✓Framework Selection Reality: Approximately 80% of successful production agents use custom-built implementations rather than established agent frameworks. Teams achieve better results by using frontier models initially, building small evaluation datasets first, and optimizing for functionality before cost. Token costs decrease roughly 90% year-over-year, making premature optimization counterproductive when establishing baseline agent performance.
What It Covers
Gideon Mendels, CEO of Comet, explains how LLM-powered agents require new evaluation and optimization approaches that blend software engineering and ML practices. He covers building evaluation datasets from production failures, using LLMs to automatically optimize prompts through search algorithms, and creating continuous improvement loops for agents in production environments.
Key Questions Answered
- •Evaluation Dataset Bootstrap Strategy: Build evaluation datasets by capturing production failures and user complaints rather than creating synthetic data upfront. When users report incorrect agent responses, document the input, expected output, and context as test cases. Starting with just 20 real-world samples provides enough foundation to run optimization algorithms and prevent regressions, making evals practical rather than theoretical.
- •Prompt Optimization as Search Problem: Treat system prompts, tool descriptions, and configurations as hyperparameters in a search space. Algorithms like JEPPA use LLMs to analyze failed test cases, suggest new prompt candidates, and iteratively improve performance. LangChain's JSON schema prompt improved from 12% to 96% accuracy in two iterations for under one dollar in API costs, demonstrating rapid, cost-effective optimization.
- •Configuration Management Over Version Control: Store prompts and agent configurations in a centralized registry rather than embedding them in code repositories. Applications fetch current configurations at runtime, enabling product managers to update prompts without redeployment, support AB testing across traffic percentages, and implement canary deployments. This separates agent behavior updates from application deployment cycles.
- •End-to-End Testing Priority: Start evaluation efforts with system-level tests that validate complete agent workflows before building unit tests for individual components. The easiest high-value evaluation checks whether agents call the correct tools given specific contexts, essentially treating tool selection as a classification problem. This provides broad coverage without requiring detailed graph traversal validation.
- •Framework Selection Reality: Approximately 80% of successful production agents use custom-built implementations rather than established agent frameworks. Teams achieve better results by using frontier models initially, building small evaluation datasets first, and optimizing for functionality before cost. Token costs decrease roughly 90% year-over-year, making premature optimization counterproductive when establishing baseline agent performance.
Notable Moment
Mendels reveals that most teams building agents skip evaluation datasets entirely, relying on manual testing of a few inputs before production deployment. This vibe-checking approach explains why fewer production agents exist than expected, as teams lack the systematic validation needed to confidently ship updates to nondeterministic systems that behave unpredictably.
Episode Transcript
LLM powered systems continue to move steadily into production, but this process is presenting teams with challenges that traditional software practices don't commonly encounter. Models and agents are non deterministic systems, which makes it difficult to test changes, reason about failures, and confidently ship updates. This has created the need for new evaluation tooling designed specifically around the properties of LLMs. COMET is a platform with roots and MLOps that has evolved to support teams building modern LLM powered applications. The company recently launched OPIC, which is an open source platform focused on evaluation, optimization, and observability for LLM agents. Together, the tools aim to bring the rigor of traditional engineering and ML workflows to the rapidly evolving world of agent based systems by treating prompts, tools, and workflows as optimizable components that can be evaluated and improved over time. Gideon Mendels is the cofounder and CEO of Comet. He previously worked at Google on hate speech and deception detection, and he founded GroupWise, which trained and deployed NLP models processing billions of chats. In this episode, Gideon joins Kevin Ball to discuss how agent development sits between software engineering and ML, why evals are the missing foundation for most AI teams, prompt optimization as a search problem, and the future for continuously improving agents in production. Kevin Ball or Kay Ball is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He cofounded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow Keball on Twitter or LinkedIn, or visit his website, keball.llc. Gideon, welcome to the show. Yeah. Kevin, thanks for having me. I'm a big fan of the podcast, so I was looking forward for this one. Yeah. I'm excited. Well, let's start with you. So can you give a little bit about your background, how you ended up at Comet, and then some of what Comet is about? Absolutely. So I'm originally, I started as a software engineer, kinda moved throughout the stack in the first kinda few years. And then about ten, twelve years ago, I shifted to working on machine learning. I was a grad student, and then I went to Google. Funny enough, I worked on language models. This is, you know, 2016, so they weren't large nor very good. Right? It's like pre transformer days, unfortunately, LCMs, if anyone still knows what that is. And, you know, as someone coming from a software engineer background where, you know, we take a lot of pride of how we build software, obviously, a lot of that changing right now. I'm sure we'll talk about it. But, you know, a lot of pride of how we build software, the tools that we use, and then joining an ML team with amazing, very, very smart and talented people. But just seeing how the whole thing is kinda like …
Get the full transcript (10,544 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 49-minute episode.
Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Software Engineering Daily
SED News: The NVIDIA-Hugging Face Deal, China’s Proxy Economy, the Open Weight Surge
Sep 8 · 52 min
The TWIML AI Podcast
How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
May 7
More from Software Engineering Daily
Moving Beyond RAG with Precomputed Context
Sep 3 · 55 min
Odd Lots
Why Laser Beams Are the Hottest New Tech in Defense
Sep 4
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
“LangChain's JSON schema prompt improved from 12% to 96% accuracy in two iterations for under one dollar in API costs, demonstrating rapid, cost-effective optimization.”
company
“Gideon Mendels, CEO of Comet, explains how LLM-powered agents require new evaluation and optimization approaches that blend software engineering and ML practices.”
More from Software Engineering Daily
We summarize every new episode. Want them in your inbox?
SED News: The NVIDIA-Hugging Face Deal, China’s Proxy Economy, the Open Weight Surge
Moving Beyond RAG with Precomputed Context
The Death of Online Anonymity
TypeScript 7 and What Comes Next
The Gap Between AI Spending and AI Value
Similar Episodes
Related episodes from other podcasts
The TWIML AI Podcast
May 7
How to Find the Agent Failures Your Evals Miss with Scott Clark - #767
Odd Lots
Sep 4
Why Laser Beams Are the Hottest New Tech in Defense
Machine Learning Street Talk
Jul 1
The Benchmark With No Instructions — ARC-AGI-3 (winning team!)
The TWIML AI Podcast
Jun 16
Why AI Agents Break the GenAI Security Model with Devvret Rishi - #770
a16z Podcast
Jun 6
Building Search for AI Agents with Exa CEO Will Bryk
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Software Engineering Daily.
Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime