Vespa AI and Surpassing the Limits of Vector Search
Episode
38 min
Read time
2 min
Topics
Productivity, Artificial Intelligence, Software Development
AI-Generated Summary
Key Takeaways
- ✓Hybrid search outperforms vectors alone: Combining BM25 lexical search with embedding models consistently outperforms either approach in isolation. Even though most modern embedding models individually beat BM25 off-the-shelf, hybrid search surpasses the models themselves. Production systems should implement both signals rather than defaulting to vector similarity as the sole relevance measure.
- ✓Tensor-based ranking enables future-proof retrieval: Representing data as tensors rather than flat vectors allows Vespa to natively support new retrieval techniques like ColPali multi-vector search and Bayesian BM25 normalization without architectural rewrites. Practitioners should model ranking signals as named tensor dimensions to enable fast dot-product operations instead of slower scripted field calculations.
- ✓Multi-stage re-ranking on content nodes reduces latency bottlenecks: Vespa executes first-phase ranking across all documents and second-phase re-ranking on top-N results directly on content nodes, avoiding expensive data movement. A third global re-ranking phase on a stateless GPU layer handles complex models. This architecture allows more sophisticated models to run within acceptable latency budgets.
- ✓Chunking strategy directly impacts vector relevance quality: Compressing an entire book or long document into one vector creates a lossy representation that loses specificity. Models like ColPali address PDF complexity by generating one vector per 32x32 patch across each page, enabling precise retrieval of specific tables or graphs within documents when text and image queries share the same vector space.
- ✓Agent accuracy compounds with retrieval quality: When AI agents run 10 sequential searches each at 90% accuracy, compound success probability drops dramatically. Improving single-query retrieval precision directly multiplies aggregate agent reliability. Poor retrieval context also increases hallucination rates because models rely on whatever context is provided, making high-precision search infrastructure a prerequisite for reliable agentic systems.
What It Covers
Vespa software engineer Radu Gheorghe explains why vector similarity alone fails in production search systems, how tensor-based retrieval generalizes ranking beyond single-signal approaches, and where multi-stage re-ranking architectures create efficiency trade-offs in RAG pipelines and AI agent workflows.
Key Questions Answered
- •Hybrid search outperforms vectors alone: Combining BM25 lexical search with embedding models consistently outperforms either approach in isolation. Even though most modern embedding models individually beat BM25 off-the-shelf, hybrid search surpasses the models themselves. Production systems should implement both signals rather than defaulting to vector similarity as the sole relevance measure.
- •Tensor-based ranking enables future-proof retrieval: Representing data as tensors rather than flat vectors allows Vespa to natively support new retrieval techniques like ColPali multi-vector search and Bayesian BM25 normalization without architectural rewrites. Practitioners should model ranking signals as named tensor dimensions to enable fast dot-product operations instead of slower scripted field calculations.
- •Multi-stage re-ranking on content nodes reduces latency bottlenecks: Vespa executes first-phase ranking across all documents and second-phase re-ranking on top-N results directly on content nodes, avoiding expensive data movement. A third global re-ranking phase on a stateless GPU layer handles complex models. This architecture allows more sophisticated models to run within acceptable latency budgets.
- •Chunking strategy directly impacts vector relevance quality: Compressing an entire book or long document into one vector creates a lossy representation that loses specificity. Models like ColPali address PDF complexity by generating one vector per 32x32 patch across each page, enabling precise retrieval of specific tables or graphs within documents when text and image queries share the same vector space.
- •Agent accuracy compounds with retrieval quality: When AI agents run 10 sequential searches each at 90% accuracy, compound success probability drops dramatically. Improving single-query retrieval precision directly multiplies aggregate agent reliability. Poor retrieval context also increases hallucination rates because models rely on whatever context is provided, making high-precision search infrastructure a prerequisite for reliable agentic systems.
Notable Moment
Radu describes how Vespa's tensor framework supported ColPali multi-vector retrieval from day one of the model's release — not because Vespa anticipated it, but because the underlying mathematical plumbing for mapping patch IDs to vectors and computing MaxSim was already in place.
Episode Transcript
Vector search has risen to become a foundational tool in modern search and retrieval systems, including the RAG pipelines that power many AI applications. However, the demands on retrieval systems are growing more sophisticated, which is revealing the limits of relying on a single vector similarity score. Vespa is a popular open source search and data serving engine. Central to Vespa's architecture is tensor based retrieval, which is an approach that represents data as tensors rather than simple vectors. Tensor based retrieval enables richer mathematical operations and more flexible ranking functions that can surmount the limitations of a single vector similarity score. Radu Gheorghe is a software engineer at Vespa with a background spanning nearly twelve years of consulting and training on Elasticsearch and Solr. In this episode, Radu joined Sean Falconer to discuss why vector similarity alone falls short in production, how tensor based retrieval generalizes to support richer ranking functions, the trade offs in chunking and multi stage re ranking architectures, and where AI search is headed next. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. Radu, welcome to the show. Hi. Thanks for having me. Yeah. Absolutely. I'm glad you're able to be here. I interviewed your founder, Vespa, and CEO probably a couple years ago, so it's great to catch up again on everything that's happening over at Vespa. A lot has changed in the world of AI and, I'm sure, in the world of Vespa over the last couple years. Yep. So you've been working in this space for a while. I I guess, like, what's your origin story? How did you end up working in search infrastructure and ultimately get involved at Vespa? Yeah. This was two jobs ago working at an antivirus company, and we needed to centralize logs. And that's how we got into Elasticsearch. And then I moved on to a company that was, at the time, at least, doing mostly consulting on top of Elasticsearch and Solr. And so I've been doing for almost twelve years at that company consulting, training, that sort of stuff for Elasticsearch and Solr and then OpenSearch. And then what ultimately led you to Vespa? Well, I guess mostly curiosity because Vespa not being based on Lucene and having different internals, different distributor model, different trade offs that it makes. Got me intrigued, met a bunch of people at conferences, was more and more curious. And, yeah, that's how I got into it. Yeah. I mean, Vespa is a company at least the origins of it has been around for a long time. Over twenty years has been working on search related problems. Can you share a little bit about some of the origins of the company and what was their original problem that they were focused on, and how much of that originating DNA is there today? I know quite a few things, but only from other people because I …
Get the full transcript (6,867 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 35-minute episode.
Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Software Engineering Daily
SED News: The Kimi Moment, Runaway AI, and Tokenmaxxing
Aug 11 · 48 min
In Good Company with Nicolai Tangen
HIGHLIGHTS: Sridhar Ramaswamy - CEO of Snowflake
Jun 19
More from Software Engineering Daily
The Terminal as an Agentic Interface
Aug 6 · 52 min
In Good Company with Nicolai Tangen
Snowflake CEO: Scaling Data, AI Agents and the New Software Era
Jun 17
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
“Tensor-based ranking enables future-proof retrieval: Representing data as tensors rather than flat vectors allows Vespa to natively support new retrieval techniques like ColPali multi-vector search and Bayesian BM25 normalization.”
- VespaBy guest
“Vespa software engineer Radu Gheorghe explains why vector similarity alone fails in production search systems... Vespa executes first-phase ranking across all documents and second-phase re-ranking on top-N results directly on content nodes.”
“Hybrid search outperforms vectors alone: Combining BM25 lexical search with embedding models consistently outperforms either approach in isolation.”
More from Software Engineering Daily
We summarize every new episode. Want them in your inbox?
SED News: The Kimi Moment, Runaway AI, and Tokenmaxxing
The Terminal as an Agentic Interface
AI-Powered Threats to the Software Supply Chain
Docker and Sandboxing AI Agents
The Startup Scene in Southeast Asia
Similar Episodes
Related episodes from other podcasts
In Good Company with Nicolai Tangen
Jun 19
HIGHLIGHTS: Sridhar Ramaswamy - CEO of Snowflake
In Good Company with Nicolai Tangen
Jun 17
Snowflake CEO: Scaling Data, AI Agents and the New Software Era
How I AI
Jun 15
How Braintrust uses AI agents, evals, and CI to ship better software | Ankur Goyal
What Bitcoin Did
Mar 10
#155 - Connor Leahy - "We Don't Know How It Works": An AI Engineer's Warning
Eye on AI
Feb 27
#324 Sharon Zhou: Inside AMD's Plan to Build Self-Improving AI
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's AI & Machine Learning Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Software Engineering Daily.
Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime