Unlocking the Data Layer for Agentic AI with Simba Khadder

What It Covers

Simba Khadder, AI strategy lead at Redis and former FeatureForm cofounder, explains why agentic AI systems require a "context engine" architecture built on four pillars: on-demand retrieval, current data, fast access, and memory that improves over time, replacing traditional RAG approaches.

Key Questions Answered

• •Context Engine Architecture: Build agentic data layers around four pillars — on-demand context retrieval, always-current data, sub-second retrieval speed, and self-improving memory. Giving agents tool-based access to navigate context dynamically outperforms preloading context windows upfront, especially as autonomous task horizons extend beyond one hour of unsupervised operation.
• •Materialized Views Over Direct DB Access: Never give agents direct access to production databases like Postgres. Instead, use ETL synchronization tools (Redis uses RDI) to build materialized views with a semantic layer on top, then compile that schema into MCP endpoints or CLI tools the agent can call safely and at scale.
• •Async Memory Compaction: Memory systems should run asynchronously alongside agents, extracting facts and compacting conflicting or outdated information using LLM-style transformations. Define custom extraction prompts targeting specific memory types — decisions made, user preferences, error patterns — rather than treating all memories as equivalent flat storage.
• •Agent Horizon Doubling Every Six Months: Anthropic's benchmark shows agents can currently complete unsupervised software tasks up to one hour in complexity, with that figure doubling every six months. Teams building RAG-only pipelines hit a hard ceiling because static context injection cannot sustain multi-hour autonomous workflows requiring dynamic, evolving information access.
• •Spec-Driven Development with Behavior Tests: Engineering teams using agents should front-load architecture reviews and write plain-English specs defining interfaces and acceptance criteria before generating code. End-to-end behavior tests covering happy paths and error cases become the primary quality gate — if behavior tests pass against agreed interfaces, the implementation is considered correct.