What are the key takeaways from this Software Engineering Daily episode?

Key insights include: **Context Window Degradation:** Chroma's ContextRot paper tested 18 models and found sharp performance drop-offs beginning in the tens of thousands of tokens. This degradation persists across three subsequent model generations, meaning developers should treat context window management as a core engineering discipline, not an afterthought, when building any RAG or agentic system.; **Agentic Query Load:** Agents issue queries at machine speed — one query every one to two seconds, with five to six parallel sub-queries each time — compared to a human's maximum of one query every thirty seconds. Retrieval infrastructure must be architected for stateless, horizontally scalable reads using shared storage layers to absorb this elastic demand without cost explosion.; **ContextOne Training Approach:** The 20B parameter ContextOne model uses a two-phase training pipeline: supervised fine-tuning on synthetic and open-weights-distilled data first, then reinforcement learning with a staged curriculum starting from one-to-two-hop problems. Teaching parallel tool-calling via SFT alone — before any RL — produces a measurable accuracy jump on agentic search benchmarks.

What did Hammad Bashir discuss on Software Engineering Daily?

Chroma CTO Hammad Bashir explains how ChromaDB evolved from a computer vision active learning tool into an AI retrieval platform, covering the ContextRot research findings on model degradation, the architecture of ContextOne (a 20B parameter retrieval sub-agent), and where intelligent data infrastructure is heading. Key topics include: **Context Window Degradation:** Chroma's ContextRot paper tested 18 models and found sharp performance drop-offs beginning in the tens of thousands of tokens. This degradation persists across three subsequent model generations, meaning developers should treat context window management as a core engineering discipline, not an afterthought, when building any RAG or agentic system.; **Agentic Query Load:** Agents issue queries at machine speed — one query every one to two seconds, with five to six parallel sub-queries each time — compared to a human's maximum of one query every thirty seconds. Retrieval infrastructure must be architected for stateless, horizontally scalable reads using shared storage layers to absorb this elastic demand without cost explosion..

How long is this episode of Software Engineering Daily?

This episode is 52 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Software Engineering Daily

Chroma and Agentic Retrieval

September 24, 2026

52 min episode · 2 min read

Hammad Bashir

Episode

52 min

Read time

2 min

Topics

Fundraising & VC, Leadership, Design & UX

AI-Generated Summary

Published Sep 24, 2026

Key Takeaways

✓Context Window Degradation: Chroma's ContextRot paper tested 18 models and found sharp performance drop-offs beginning in the tens of thousands of tokens. This degradation persists across three subsequent model generations, meaning developers should treat context window management as a core engineering discipline, not an afterthought, when building any RAG or agentic system.
✓Agentic Query Load: Agents issue queries at machine speed — one query every one to two seconds, with five to six parallel sub-queries each time — compared to a human's maximum of one query every thirty seconds. Retrieval infrastructure must be architected for stateless, horizontally scalable reads using shared storage layers to absorb this elastic demand without cost explosion.
✓ContextOne Training Approach: The 20B parameter ContextOne model uses a two-phase training pipeline: supervised fine-tuning on synthetic and open-weights-distilled data first, then reinforcement learning with a staged curriculum starting from one-to-two-hop problems. Teaching parallel tool-calling via SFT alone — before any RL — produces a measurable accuracy jump on agentic search benchmarks.
✓Self-Pruning Context Management: ContextOne includes a "prune chunks" tool allowing the model to delete documents from its own context window mid-search, keeping the active window near 32,000 tokens. Pruning accuracy reached 95%+ after training (versus 80% at baseline), reducing prefill and decode costs enough to sustain 400–500 tokens per second versus 40–80 for frontier models like Opus.
✓BYOC Architecture Requirement: Bring-your-own-cloud deployment must be designed from day one — retrofitting it later is described as nearly impossible. The viable pattern uses no open inbound ports on the data plane; instead, a service inside the customer VPC polls outbound for operations and pushes telemetry, keeping all private data within the customer's infrastructure while the control plane remains with Chroma.

What It Covers

Chroma CTO Hammad Bashir explains how ChromaDB evolved from a computer vision active learning tool into an AI retrieval platform, covering the ContextRot research findings on model degradation, the architecture of ContextOne (a 20B parameter retrieval sub-agent), and where intelligent data infrastructure is heading.

Key Questions Answered

•Context Window Degradation: Chroma's ContextRot paper tested 18 models and found sharp performance drop-offs beginning in the tens of thousands of tokens. This degradation persists across three subsequent model generations, meaning developers should treat context window management as a core engineering discipline, not an afterthought, when building any RAG or agentic system.
•Agentic Query Load: Agents issue queries at machine speed — one query every one to two seconds, with five to six parallel sub-queries each time — compared to a human's maximum of one query every thirty seconds. Retrieval infrastructure must be architected for stateless, horizontally scalable reads using shared storage layers to absorb this elastic demand without cost explosion.
•ContextOne Training Approach: The 20B parameter ContextOne model uses a two-phase training pipeline: supervised fine-tuning on synthetic and open-weights-distilled data first, then reinforcement learning with a staged curriculum starting from one-to-two-hop problems. Teaching parallel tool-calling via SFT alone — before any RL — produces a measurable accuracy jump on agentic search benchmarks.
•Self-Pruning Context Management: ContextOne includes a "prune chunks" tool allowing the model to delete documents from its own context window mid-search, keeping the active window near 32,000 tokens. Pruning accuracy reached 95%+ after training (versus 80% at baseline), reducing prefill and decode costs enough to sustain 400–500 tokens per second versus 40–80 for frontier models like Opus.
•BYOC Architecture Requirement: Bring-your-own-cloud deployment must be designed from day one — retrofitting it later is described as nearly impossible. The viable pattern uses no open inbound ports on the data plane; instead, a service inside the customer VPC polls outbound for operations and pushes telemetry, keeping all private data within the customer's infrastructure while the control plane remains with Chroma.

Notable Moment

Chroma's CEO of Databricks reference cuts against BYOC enthusiasm: the Databricks CEO reportedly said he would skip the hybrid deployment model entirely if rebuilding. Chroma built it from day one anyway, arguing it was structurally necessary to serve large-scale enterprise customers — but warns it severely slows early development velocity.

Know someone who'd find this useful?

Episode Transcript

Are you passionate about software development and the tech industry? Software Engineering Daily is looking for a new podcast host to grow its hosting team. In this role, you'll help shape the show's editorial direction and interview engineers, founders, hackers, and tech leaders. Podcasting experience is a plus, but not required. Curiosity, great communication skills, and a genuine interest in the craft of building software are what matter most. If this sounds like you, reach out at editor@softwareengineeringdaily.com. Data retrieval is a fundamental challenge in AI systems, and the approaches for solving it are still evolving. Vector search was an early answer to the retrieval problem, but the rise of agentic systems has raised the stakes considerably. Agents issue queries at machine speed, decompose complex questions into parallel searches, and require retrieval infrastructure that can keep pace without becoming prohibitively expensive. Chroma is a company building open source infrastructure for AI applications, best known for its widely used database of the same name. The company also published the influential ContextRot paper, which documented how model performance degrades as context window utilization increases, and recently released ContextOne, a 20,000,000,000 parameter retrieval sub agent trained to do agentic search at frontier model quality but at an order of magnitude lower cost and higher speed. Hammad Bashir is the CTO of Chroma. With a background spanning machine learning, computer vision, and data systems. In this episode, Hammad joins Gregor Van to discuss the origins of ChromaDB, our current understanding of context rot, why a purpose built small model can match frontier models on search tasks, the philosophy behind Chroma's open source approach, and where the company sees AI data infrastructure heading. Gregor Vand is a security focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at vand.hk or on LinkedIn. Hello, welcome to Software Engineering Daily. My guest today is Hammad Bashir from Chroma. So nice to have you. Yeah. Great to be here. Yeah. Awesome. So we're gonna be hearing all about Chroma. I think it's a product that probably a lot of our audience are familiar with and probably familiar with these types of products as well. But before we get into that, just love to hear as we like to, how about you've worked at quite a few interesting companies, I think in the Bay Area and beyond. What's your quick path to getting to Chroma and all that? Yeah. I spent better part of the decade as a software engineer and broad expertise in machine learning, computer vision and data systems was most of what I worked on for the past decade. And before this was spending a lot of time specifically on computer vision. And initially when we started on Chroma, the problem that we were solving is one of active learning for computer vision systems, which is the problem of every day you have a bunch of live …

Get the full transcript (10,696 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →