Open-Weight AI Models
Episode
50 min
Read time
2 min
Topics
Investing, Startups, Fundraising & VC
AI-Generated Summary
Key Takeaways
- ✓Open-weight model selection: Roughly one-third of Fireworks customers arrive knowing exactly which model to deploy; one-third need cost and scalability guidance between two or three candidates; one-third rely fully on Fireworks evaluations. Knowing which category you fall into determines how much internal ML expertise you need before engaging an inference platform.
- ✓Speculative decoding for production: Fireworks trains custom speculator draft models specifically matched to each customer's fine-tuned target model, not generic open-source speculators. This pairing is critical for latency-sensitive workloads like Cursor's fast-apply feature, where a large file must be edited in one pass at high speed and low cost.
- ✓Reinforcement fine-tuning unlocks non-ML teams: RFT removes the need for MLE-managed data labeling pipelines. A product manager who can articulate what "good output" looks like can author a language-model-as-judge eval, send it to Fireworks, and trigger a training run. Vercel used this approach with two to three people and achieved 40x faster code-fixing with improved output quality.
- ✓Evals as compounding business assets: Unlike supervised fine-tuning datasets that require curation updates as models evolve, RL evaluation environments remain valid across model generations. Building evals now via Fireworks' open-source eval-protocol framework means the same asset used to benchmark models today can directly drive RFT training runs tomorrow without significant rework.
- ✓Multi-hardware supply chain strategy: Running inference across both NVIDIA and AMD hardware is primarily a supply chain reliability decision, not a performance one. At peak demand, NVIDIA cards become unavailable at reasonable prices, so maintaining AMD kernel support through custom in-house fire-attention kernels ensures uninterrupted capacity and competitive pricing for customers.
What It Covers
Fireworks AI cofounder Benny Chen explains how his company serves and customizes open-weight models at scale, processing over 13 trillion tokens daily. The episode covers custom inference kernels, speculative decoding, multi-hardware strategy across NVIDIA and AMD, reinforcement fine-tuning, and why evals represent a durable business asset.
Key Questions Answered
- •Open-weight model selection: Roughly one-third of Fireworks customers arrive knowing exactly which model to deploy; one-third need cost and scalability guidance between two or three candidates; one-third rely fully on Fireworks evaluations. Knowing which category you fall into determines how much internal ML expertise you need before engaging an inference platform.
- •Speculative decoding for production: Fireworks trains custom speculator draft models specifically matched to each customer's fine-tuned target model, not generic open-source speculators. This pairing is critical for latency-sensitive workloads like Cursor's fast-apply feature, where a large file must be edited in one pass at high speed and low cost.
- •Reinforcement fine-tuning unlocks non-ML teams: RFT removes the need for MLE-managed data labeling pipelines. A product manager who can articulate what "good output" looks like can author a language-model-as-judge eval, send it to Fireworks, and trigger a training run. Vercel used this approach with two to three people and achieved 40x faster code-fixing with improved output quality.
- •Evals as compounding business assets: Unlike supervised fine-tuning datasets that require curation updates as models evolve, RL evaluation environments remain valid across model generations. Building evals now via Fireworks' open-source eval-protocol framework means the same asset used to benchmark models today can directly drive RFT training runs tomorrow without significant rework.
- •Multi-hardware supply chain strategy: Running inference across both NVIDIA and AMD hardware is primarily a supply chain reliability decision, not a performance one. At peak demand, NVIDIA cards become unavailable at reasonable prices, so maintaining AMD kernel support through custom in-house fire-attention kernels ensures uninterrupted capacity and competitive pricing for customers.
Notable Moment
Chen noted that Fireworks launched roughly five to six months before ChatGPT shipped, betting on open-weight models when the best available options could barely sustain a three-turn conversation and had no function-calling capability — a position he describes as genuinely contrarian at the time.
Episode Transcript
Open weight models are AI systems whose trained parameters are publicly released, which allows developers to run, fine tune, and deploy them independently rather than accessing them only through a hosted API. While closed weight models from companies like OpenAI or Anthropic are delivered as managed services, open weight models give organizations direct control over how the models are deployed and used. Importantly, the performance of these models is steadily improving, and they've become credible alternatives for production workloads with advantages and customization and data privacy. Fireworks AI is building a platform focused on serving and customizing open weight models at scale. The platform includes optimized inference infrastructure, multi hardware support across NVIDIA and AMD, and reinforcement fine tuning capabilities. Benny Chen is a cofounder of Fireworks AI. In this episode, he joins Gregor Van to discuss his path from Meta's ML infrastructure teams to cofounding Fireworks AI, why open weight models are becoming increasingly competitive, how custom kernels and speculative decoding improve performance, reinforcement fine tuning, and much more. Gregor Vand is a security focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn. Hello, and welcome to Software Engineering Daily. My guest today is Benny Chen. Welcome, Benny. Thanks for having me. Yeah. Great to have you here. So we're gonna be talking all about Fireworks AI, which is a company that I believe you cofounded, I think, three, three and a half years ago. Is that right? Yeah. Nice. Yeah. So before we dive into FireX AI, and I think, especially today, this is quite pertinent to sort of where maybe FireX AI came from. You spent a lot of time at Meta and, you know, in their ML team, which means you were doing things with ML probably way before many of us really had it on our radar. But what was kinda your path? Like, what's been your path to like, through software engineering and and especially the Meta phase as well? Yeah. That that's definitely, it's been a while. So let me what the journey was like in the beginning. So in the very beginning, I joined as a software engineer in 2014 on the integrity team where most of, I would say, the so, like, known recommendation system experiments started. Early on, it was sort of, like, decision trees for different fraud behaviors. And then also on the team, we started doing, like, image classifiers for different rules for advertising. And then come 2016, switched over to the ads infrastructure, so worked on supporting the recommendation system models. And then in 2017, I think the Facebook leadership back then started thinking about having a ASIC in house that's more like Google's TPU. So we started collaborating with Intel back then on a, ASIC for recommendation systems. So is this anything about, like, doing something that's not cool before everyone remembers? …
Get the full transcript (9,058 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 47-minute episode.
Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Software Engineering Daily
A Rust Framework to Simplify Distributed Systems
Sep 10 · 50 min
The Lean Startup
How GitLab scaled to 30M users with transparency, remote work, and the ultimate employee handbook | Sid Sijbrandij
Jul 10
More from Software Engineering Daily
SED News: The NVIDIA-Hugging Face Deal, China’s Proxy Economy, the Open Weight Surge
Sep 8 · 52 min
Masters of Scale
The quiet reinvention of a $42b business, with Canva’s Cameron Adams
Jul 14
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.
Tools
by Fireworks AI
“Building evals now via Fireworks' open-source eval-protocol framework means the same asset used to benchmark models today can directly drive RFT training runs tomorrow without significant rework.”
Products
by Cursor
“This pairing is critical for latency-sensitive workloads like Cursor's fast-apply feature, where a large file must be edited in one pass at high speed and low cost.”
company
- Fireworks AIBy guest
“Fireworks AI cofounder Benny Chen explains how his company serves and customizes open-weight models at scale, processing over 13 trillion tokens daily.”
More from Software Engineering Daily
We summarize every new episode. Want them in your inbox?
A Rust Framework to Simplify Distributed Systems
SED News: The NVIDIA-Hugging Face Deal, China’s Proxy Economy, the Open Weight Surge
Moving Beyond RAG with Precomputed Context
The Death of Online Anonymity
TypeScript 7 and What Comes Next
Similar Episodes
Related episodes from other podcasts
The Lean Startup
Jul 10
How GitLab scaled to 30M users with transparency, remote work, and the ultimate employee handbook | Sid Sijbrandij
Masters of Scale
Jul 14
The quiet reinvention of a $42b business, with Canva’s Cameron Adams
David Senra
Jul 12
Eric Glyman, Co-founder of Ramp
In Good Company with Nicolai Tangen
Jun 17
Snowflake CEO: Scaling Data, AI Agents and the New Software Era
Eye on AI
Jun 6
Every Enterprise Is About to Have a 100,000 Agent Problem | Oren Michaels of Barndoor AI
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Software Engineering Daily.
Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime