Is RAG Dead? Lessons from Building AI for Tax Law with Alex Bowcut - #769

As context windows get larger and larger, one question that keeps coming up is whether retrieval augmented generation or RAG is becoming obsolete. If models can ingest millions of tokens of context and reason over enormous collections of documents, why bother with retrieval at all? The answer, it turns out, depends a lot on the application. I recently sat down with Alex Bocut, head of engineering at Sphere, which builds AI systems for sales tax automation and compliance. Exactly the kind of domain where getting the right answer isn't enough. You also need to know where it came from. I asked them this simple question. What's your take on the whole rag is dead argument that some folks make? I think for some use cases, it's certainly true. I think for us and or at least for this particular problem, because we are so sensitive to accuracy and we're so sensitive to the exact right citation, as of today, I don't think, you know, agents are just searching over the file system, grepping over it. Is that a point where we could switch over and and not lose accuracy? I'm Sam Charrington, and this is the TwiML AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in. A little bit about Sphere briefly so so this makes a little more sense. Sphere Sphere is a a revenue based compliance company. So we help companies with all of their revenue based compliance needs. The main one of those is is sales tax, in The US and internationally. That's called VATGST. And the way you know, there there's other companies in this space, of course. There's some big companies that have been around for quite a while. Tax is not a new problem. It's it's not, unfortunately, for companies and for consumers, I suppose. So this isn't a new problem. The incumbents, they face a particular problem, which is in order to support, you know, every jurisdiction in The US, because every, of course, every US state has different rules. In some US states, even the cities have different rules. And then internationally, of of course, every country and potentially province has their own rules as well. And so the companies need, the incumbents need a way to understand how our products taxed in each of these different jurisdictions. And the way that, traditionally, they've done these sorts of things is they've hired these massive teams of essentially tax lawyers. They're tax experts. They'll call them tax content teams. What these tax lawyers are doing essentially is looking through the legislation in, you know, Alabama, for example, and understanding how does Alabama tax SaaS. And even more specifically than that, how does Alabama tax SaaS that maybe has an API connection and has servers that are hosted within the state itself. So it it gets …