Building Open Infrastructure for AI with Illia Polosukhin

Ilya Polosekin is a veteran AI researcher and one of the original authors of the landmark transformer paper, Attention is All You Need, which he coauthored during his time at Google Research. He has a deep background in machine learning and natural language processing and has processing and has spent over a decade working at the intersection of AI and decentralized technologies. His current venture is called Near AI, and he's focused on building open source infrastructure, tools, and products for agentic privacy preserving AI systems. He joins the podcast with Kevin Ball to discuss his journey, the origins of the transformer model, the vision for user owned AI, document oriented development, and much more. Kevin Ball or Kay Ball is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He cofounded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow Kay Ball on Twitter or LinkedIn, or visit his website, kball.llc. Ilya, welcome to the show. Thanks for having me. Yeah. Excited to get to talk with you. Let's maybe start with a little bit of intro about you, your background, and what you're up to these days. For sure. Yeah. Well, I've been, I guess, tech geek since I was 10 years old. Been building a lot of video games back in the day and then got really excited about machine learning when I was like well, about AI in general and then started learning machine learning when I was, like, 14. I was building my first neural networks in Pascal and got a job actually remotely. So I'm emerging from Ukraine working for this machine learning company out of San Diego. And they were happy with my work, and so they offered me to move. I moved to US, which was exciting. And then I saw the cat neuron paper that came out from Google from Andreang and Jeff Dean. And I was like, okay. This is the thing. Like, the unsupervised pretraining, learning about concepts in the world. They don't need supervision. And so I was like, okay. I wanna do that. And so I applied. I got into Google Research, and I always thought that, yes, images are cool, but there's thousands of species that can see. But there's only one, maybe some people argue, maybe two, that can actually speak. And, like, language is affected a way we test intelligence. Right? We ask questions. You ask a person to read the text, and we ask questions if they understand it. And so that's why I wanted to focus on natural language. We were doing question answering, trying to build products into google.com where when you ask a question, it would give you a response. This is where your previous, I guess, CTO, right, and was my director back in Google. And one of the challenges …