Skip to main content
Software Engineering Daily

Unlocking the Data Layer for Agentic AI with Simba Khadder

49 min episode · 2 min read
·
Simba Khadder

Episode

49 min

Read time

2 min

Topics

Remote Work, Startups, Fundraising & VC

AI-Generated Summary

Key Takeaways

  • Context Engine Architecture: Build agentic data layers around four pillars — on-demand context retrieval, always-current data, sub-second retrieval speed, and self-improving memory. Giving agents tool-based access to navigate context dynamically outperforms preloading context windows upfront, especially as autonomous task horizons extend beyond one hour of unsupervised operation.
  • Materialized Views Over Direct DB Access: Never give agents direct access to production databases like Postgres. Instead, use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top, then compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale.
  • Async Memory Compaction: Memory systems should run asynchronously alongside agents, extracting facts and compacting conflicting or outdated information using LLM-style transformations. Define custom extraction prompts targeting specific memory types — decisions made, user preferences, error patterns — rather than treating all memories as equivalent flat storage.
  • Agent Horizon Doubling Every Six Months: Anthropic's benchmark shows agents can currently complete unsupervised software tasks up to one hour in complexity, with that figure doubling every six months. Teams building RAG-only pipelines hit a hard ceiling because static context injection cannot sustain multi-hour autonomous workflows requiring dynamic, evolving information access.
  • Spec-Driven Development with Behavior Tests: Engineering teams using agents should front-load architecture reviews and write plain-English specs defining interfaces and acceptance criteria before generating code. End-to-end behavior tests covering happy paths and error cases become the primary quality gate — if behavior tests pass against agreed interfaces, the implementation is considered correct.

What It Covers

Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder, explains why agentic AI systems require a "context engine" architecture built on four pillars: on-demand retrieval, current data, fast access, and memory that improves over time, replacing traditional RAG approaches.

Key Questions Answered

  • Context Engine Architecture: Build agentic data layers around four pillars — on-demand context retrieval, always-current data, sub-second retrieval speed, and self-improving memory. Giving agents tool-based access to navigate context dynamically outperforms preloading context windows upfront, especially as autonomous task horizons extend beyond one hour of unsupervised operation.
  • Materialized Views Over Direct DB Access: Never give agents direct access to production databases like Postgres. Instead, use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top, then compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale.
  • Async Memory Compaction: Memory systems should run asynchronously alongside agents, extracting facts and compacting conflicting or outdated information using LLM-style transformations. Define custom extraction prompts targeting specific memory types — decisions made, user preferences, error patterns — rather than treating all memories as equivalent flat storage.
  • Agent Horizon Doubling Every Six Months: Anthropic's benchmark shows agents can currently complete unsupervised software tasks up to one hour in complexity, with that figure doubling every six months. Teams building RAG-only pipelines hit a hard ceiling because static context injection cannot sustain multi-hour autonomous workflows requiring dynamic, evolving information access.
  • Spec-Driven Development with Behavior Tests: Engineering teams using agents should front-load architecture reviews and write plain-English specs defining interfaces and acceptance criteria before generating code. End-to-end behavior tests covering happy paths and error cases become the primary quality gate — if behavior tests pass against agreed interfaces, the implementation is considered correct.

Notable Moment

Khadder describes how a non-technical marketing professional, with no prior coding experience, used Claude to build and deploy a fully functional data visualization website for her team — pulling from spreadsheets and rendering graphics — completing it faster than a traditional analyst workflow would allow.

Know someone who'd find this useful?

Episode Transcript

AI agents are increasingly capable of reasoning and performing autonomous work over long periods. However, as agents take on more complex, longer horizon tasks, keeping them supplied with the right information becomes the core engineering challenge. The industry is moving away from preloading context upfront toward a model where agents dynamically navigate and retrieve the data they need when they need it. Redis is approaching context management using a context engine, which is an architecture built around four pillars, on demand context retrieval, data that is always current, fast retrieval, and a memory layer that improves over time. In practice, this means building materialized views of data with the semantic layer on top, rather than giving agents direct access to production databases. A memory system sits alongside this, extracting and compacting information asynchronously as the agent works. Simba Kotter leads AI strategy at Redis, and he previously cofounded the feature store platform FeatureForm, which was acquired by Redis in 2025. In this episode, Simba joins Kevin Ball to discuss why context has become the defining challenge in Agentic AI, how context engines differ from traditional RAG architectures, how materialized views underpin reliable agent data pipelines, how memory systems can improve through async extraction and compaction, and how engineering teams need to adapt their practices as AI driven development accelerates. Kevin Ball, or Kay Ball, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He cofounded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow Keball on Twitter or LinkedIn or visit his website, keball.llc. Simba, welcome to the show. Thanks for having me. Yeah. Well, let's get started learning a little bit about you and your background, and then we can move into Redis and talking about that. So how do you explain yourself to folks? When I grow up, I wanna surf all day. But for today, I'm in AI. Okay. I mean, I could resonate with that. Surfing is a blast. Yeah. I started as a software engineer, then technical. When I started my career, I was at Google, a software engineer there. I solved a lot of really fun technical problems, worked with some super smart people, But I was always kinda itching to, like, go and learn on a slope that I felt like I had been learning on before. So I left Google. I started my first company, kind of did well there, started a company after, which was FeatureForm, which then was acquired by Redis. And I've always loved Redis as a product, so it's kind of awesome to become a huge part of the AI strategy here. But, yeah, that's kind of the the short. Alright. Well, and can you quickly describe FeatureForm and what it is so that we have that as well as background? Yeah. So FeatureForm, we were a feature …

Get the full transcript (10,160 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →

You just read a 3-minute summary of a 46-minute episode.

Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links.

Tools

  • by Redis

    use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top
  • Never give agents direct access to production databases like Postgres
  • compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale
  • by Anthropic

    a non-technical marketing professional, with no prior coding experience, used Claude to build and deploy a fully functional data visualization website

company

  • Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder
  • Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder, explains why agentic AI systems require a "context engine" architecture

More from Software Engineering Daily

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Startups & Product Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into Software Engineering Daily.

Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime