Recurrence and Attention for Long-Context Transformers with Jacob Buckman - #750

All axes of scale are important. I don't wanna say that state is the only important one at all. What what you really want is a is a architecture that's balanced, right? Just like I was saying before, it's you know, in this field, it's all about balance. That's how you know. When a model is beautiful and balanced and elegant, that's how you know it's gonna be a good compute optimal model. And the problem with having a state that's really big or really small fundamentally is that it's imbalanced. So, a way to think about this is Alright, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Jacob Buckman. Jacob is cofounder and CEO of Manifest AI. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Jacob, welcome to the podcast. Thanks so much for having me, Sam. I'm excited to meet you and dig into our conversation. We're going to be talking about achieving long context with transformers and the new power retention architecture that you recently published. To get us started, I'd love to have you share a little bit about your background. Yeah. Absolutely. So I've been working on deep learning for better part of a decade now. I started maybe in 2016, and I was an undergrad at Carnegie Mellon. I did my master's thesis on deep learning for autoregressive language modeling, basically, just because I thought it was was cool. It was exciting. And then, you know, here we are ten years later, and it's apparently the coolest and most exciting thing in the world, which was definitely not something I anticipated when I started getting into it. But, pretty much just Turns out. As it turns out. Yeah. And so I worked for Google Brain for a while, got lots of exposure to different areas of deep learning, some GAN stuff, some general adversarial networks, a little bit of adversarial examples, worked on deep reinforcement learning, and then I eventually went to go get a PhD in mostly deep reinforcement learning at, Mila in in Montreal. So, yeah, for long, I feel like the the writing was on the wall that just actually pushing the current paradigm was gonna take us to unbelievable places. And I started thinking together with my, cofounder, Carlos, how far is it gonna take us? Right? Like, people at this point were speculating, maybe scale will take us all the way to AGI. And, you know, they had a point. Right? And they had a pretty good argument. And one of the things that we started thinking about is, is it gonna take us all the way? Do we see any technical bottlenecks that will actually prevent us from getting there? And the conclusion that we reached was that although several of the major axes of scale are handled well, you know, we know …