How Open-Source AI Became Critical Infrastructure

The fun thought experiment is if GPUs dropped in price by 99%, then do we get back to a real open source world. If moderation is never solved in the future, people will go to Openwave by default because that is where you know for sure you can control your guardrail for trusted use cases. And can you talk about where VLOM sits in that stack? VLOM is a inference engine. It is kind of like databases and operating system and MD, Google. Their newest chip will make sure VEOM can run on them, and then a lot of cases they use VEOM as a benchmark. We're bridging almost a 10 x gap. For proprietary proprietary model, there is a regular mode and fast mode. But for open weight, every provider can offer potentially even 10 different levels of speed. Five years from now, open source AI models, have they closed the gap with Frontier models? Capability wise, I don't really see a big gap, not even today, because open source AI has become one of the most important forces shaping the industry. But making Frontier models available to the world requires far more than releasing model weights. It requires an entirely new infrastructure layer. In this episode, Elena Burger and Matt Borenstein are joined by Simon Moe, cofounder and CEO of InfraRack, to discuss how open source inference became critical Today, we're here with Simon Moe, cofounder of InfraAct and a lead maintainer of VLLM, the open source inference engine now running on half a million GPUs at any moment. We're also joined by Matt Borenstein, an a sixteen z general partner. Simon, Matt, thank you so much for joining us. I think first, we should start with the open source AI and kind of the more recent history of open source AI. So VLLM actually has its origins kind of back in 2022 pre Chargept, and your team set out to make a slow open source demo faster and instead just found this pile of unsolved problems. So can you talk about what made serving an LLM so fundamentally different from the ML workloads everyone already knew how to run? Yep. Good to be here. So serving a large language model is a fundamentally different problem because serving it requires to run it on accelerators like GPUs or TPUs. And it is a computationally intensive process that will require a lot of engineering and ensuring that for each request, user can see the LOM's response quickly and efficiently. So this typically means we need to handle differences in input distribution, how long each request is, output distribution, which is nondeterministic, and batching and scheduling a lot more in the at the core of the inference engine. Yeah. And so the team, the project has been around for about four years now, but the company is a little bit more recent. So I wanna throw this out to either of you. Matt, I know you've known the team and …