Unlocking the Data Layer for Agentic AI with Simba Khadder
Episode
49 min
Read time
2 min
Topics
Remote Work, Startups, Fundraising & VC
AI-Generated Summary
Key Takeaways
- ✓Context Engine Architecture: Build agentic data layers around four pillars — on-demand context retrieval, always-current data, sub-second retrieval speed, and self-improving memory. Giving agents tool-based access to navigate context dynamically outperforms preloading context windows upfront, especially as autonomous task horizons extend beyond one hour of unsupervised operation.
- ✓Materialized Views Over Direct DB Access: Never give agents direct access to production databases like Postgres. Instead, use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top, then compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale.
- ✓Async Memory Compaction: Memory systems should run asynchronously alongside agents, extracting facts and compacting conflicting or outdated information using LLM-style transformations. Define custom extraction prompts targeting specific memory types — decisions made, user preferences, error patterns — rather than treating all memories as equivalent flat storage.
- ✓Agent Horizon Doubling Every Six Months: Anthropic's benchmark shows agents can currently complete unsupervised software tasks up to one hour in complexity, with that figure doubling every six months. Teams building RAG-only pipelines hit a hard ceiling because static context injection cannot sustain multi-hour autonomous workflows requiring dynamic, evolving information access.
- ✓Spec-Driven Development with Behavior Tests: Engineering teams using agents should front-load architecture reviews and write plain-English specs defining interfaces and acceptance criteria before generating code. End-to-end behavior tests covering happy paths and error cases become the primary quality gate — if behavior tests pass against agreed interfaces, the implementation is considered correct.
What It Covers
Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder, explains why agentic AI systems require a "context engine" architecture built on four pillars: on-demand retrieval, current data, fast access, and memory that improves over time, replacing traditional RAG approaches.
Key Questions Answered
- •Context Engine Architecture: Build agentic data layers around four pillars — on-demand context retrieval, always-current data, sub-second retrieval speed, and self-improving memory. Giving agents tool-based access to navigate context dynamically outperforms preloading context windows upfront, especially as autonomous task horizons extend beyond one hour of unsupervised operation.
- •Materialized Views Over Direct DB Access: Never give agents direct access to production databases like Postgres. Instead, use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top, then compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale.
- •Async Memory Compaction: Memory systems should run asynchronously alongside agents, extracting facts and compacting conflicting or outdated information using LLM-style transformations. Define custom extraction prompts targeting specific memory types — decisions made, user preferences, error patterns — rather than treating all memories as equivalent flat storage.
- •Agent Horizon Doubling Every Six Months: Anthropic's benchmark shows agents can currently complete unsupervised software tasks up to one hour in complexity, with that figure doubling every six months. Teams building RAG-only pipelines hit a hard ceiling because static context injection cannot sustain multi-hour autonomous workflows requiring dynamic, evolving information access.
- •Spec-Driven Development with Behavior Tests: Engineering teams using agents should front-load architecture reviews and write plain-English specs defining interfaces and acceptance criteria before generating code. End-to-end behavior tests covering happy paths and error cases become the primary quality gate — if behavior tests pass against agreed interfaces, the implementation is considered correct.
Notable Moment
Khadder describes how a non-technical marketing professional, with no prior coding experience, used Claude to build and deploy a fully functional data visualization website for her team — pulling from spreadsheets and rendering graphics — completing it faster than a traditional analyst workflow would allow.
Episode Transcript
AI agents are increasingly capable of reasoning and performing autonomous work over long periods. However, as agents take on more complex, longer horizon tasks, keeping them supplied with the right information becomes the core engineering challenge. The industry is moving away from preloading context upfront toward a model where agents dynamically navigate and retrieve the data they need when they need it. Redis is approaching context management using a context engine, which is an architecture built around four pillars, on demand context retrieval, data that is always current, fast retrieval, and a memory layer that improves over time. In practice, this means building materialized views of data with the semantic layer on top, rather than giving agents direct access to production databases. A memory system sits alongside this, extracting and compacting information asynchronously as the agent works. Simba Kotter leads AI strategy at Redis, and he previously cofounded the feature store platform FeatureForm, which was acquired by Redis in 2025. In this episode, Simba joins Kevin Ball to discuss why context has become the defining challenge in Agentic AI, how context engines differ from traditional RAG architectures, how materialized views underpin reliable agent data pipelines, how memory systems can improve through async extraction and compaction, and how engineering teams need to adapt their practices as AI driven development accelerates. Kevin Ball, or Kay Ball, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He cofounded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow Keball on Twitter or LinkedIn or visit his website, keball.llc. Simba, welcome to the show. Thanks for having me. Yeah. Well, let's get started learning a little bit about you and your background, and then we can move into Redis and talking about that. So how do you explain yourself to folks? When I grow up, I wanna surf all day. But for today, I'm in AI. Okay. I mean, I could resonate with that. Surfing is a blast. Yeah. I started as a software engineer, then technical. When I started my career, I was at Google, a software engineer there. I solved a lot of really fun technical problems, worked with some super smart people, But I was always kinda itching to, like, go and learn on a slope that I felt like I had been learning on before. So I left Google. I started my first company, kind of did well there, started a company after, which was FeatureForm, which then was acquired by Redis. And I've always loved Redis as a product, so it's kind of awesome to become a huge part of the AI strategy here. But, yeah, that's kind of the the short. Alright. Well, and can you quickly describe FeatureForm and what it is so that we have that as well as background? Yeah. So FeatureForm, we were a feature …
Get the full transcript (10,160 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 46-minute episode.
Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Software Engineering Daily
Moving Beyond RAG with Precomputed Context
Sep 3 · 55 min
Bankless
Zero Crypto at Home: Bankless in the Age of Wrench Attacks and Phishing | Jameson Lopp and Beau
Feb 25
More from Software Engineering Daily
The Death of Online Anonymity
Sep 1 · 52 min
How I Built This
Advice Line with Ben Goodwin of Olipop
Sep 3
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
by Redis
“use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top”
“Never give agents direct access to production databases like Postgres”
“compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale”
by Anthropic
“a non-technical marketing professional, with no prior coding experience, used Claude to build and deploy a fully functional data visualization website”
company
“Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder”
“Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder, explains why agentic AI systems require a "context engine" architecture”
More from Software Engineering Daily
We summarize every new episode. Want them in your inbox?
Similar Episodes
Related episodes from other podcasts
Bankless
Feb 25
Zero Crypto at Home: Bankless in the Age of Wrench Attacks and Phishing | Jameson Lopp and Beau
How I Built This
Sep 3
Advice Line with Ben Goodwin of Olipop
Investing for Beginners
Aug 31
How to Get 9% Returns with Half the Market Volatility
Lenny's Podcast
Aug 30
AI’s third era: the rise of persistent AI coworkers | Tara Seshan (OpenAI’s product lead)
Masters of Scale
Aug 18
What the jobs report isn't telling you about AI & the workforce, with Upwork’s Hayden Brown
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Startups & Product Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Software Engineering Daily.
Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime