How Foundation Models Evolved: A PhD Journey Through AI's Breakthrough Era

Nobody wants intelligence period. I want something else. Right? And that something else is always specific or at least more specific. There is this kind of observed phenomenon where if you over engineer intelligence, you regret it because somebody figures out a more general and maybe potentially simpler method that scales better, and a lot of the hard coded decisions you made are things you end up regretting. So I think it's fair to assume that, like, models will get better and algorithms will get better, and a lot of that stuff will improve. Then the question we really ask is intelligence is great, but what problems are you actually trying to solve? That idea that scaling model parameters and scaling just pretraining data is all you need is exists nowhere anymore. Nobody thinks that. Actually, people deny they ever thought that at this point. Now you see this massively human designed and very carefully constructed pipelines for post training where we really encode a lot of the things we wanna do. You see massive emphasis on retrieval and web search and tool use and agent training. There is clearly a a sense in which the labs have already recognized that the old playbook doesn't work. The question is, is that actually sufficient for making the best use and the most used of of these language models? It's not a problem of capabilities. It's a problem of, actually, we don't necessarily just need models. We want systems. The conventional wisdom says we're racing toward AGI by making language models bigger and bigger. But what if the entire framing is wrong? On today's episode, you'll hear from a sixteen z general partner, Martin Casado, and guest Omar Khattab, assistant professor at MIT and creator of dspy. Omar doesn't think we need artificial general intelligence. He thinks we need artificial programmable intelligence, and the difference matters more than you think. Here's the paradox. Khattab has built one of the most widely used frameworks for working with LLMs, dspie, but he's skeptical that raw model capabilities will solve our problems. While others obsess over scaling laws and parameter counts, he's asking a more fundamental question. Even if models become infinitely scalable, infinitely capable, how do humans actually specify what they want? Natural language is too ambiguous. Code is too rigid. We need something in between, a new abstraction layer that lets us declare intent without drowning implementation details. Think of it as a jump from assembly to C or for AI systems. The stakes are higher than prompt engineering. This is about whether AI becomes a programmable tool we can reason about and compose or just an inscrutable oracle we prompt and pray. We get into the three irreducible pieces of an AI system, why the model god is a dead end, and what it actually means to build software when intelligence is cheap but the specification is hard. Well, listen, Omar, it's great to have you, and congratulations on everything. Just so …