What are the key takeaways from this Latent Space episode?

Key insights include: **Agent Experience over Developer Experience:** Modal reoriented its SDK team from developer experience to agent experience, finding that co-locating infrastructure configuration inside code decorators — rather than sprawling YAML files — lets AI agents provision and modify compute environments faster. Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows.; **Speculative Decoding via Block-Based Drafting:** Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains with zero quality degradation. Kernel-level optimizations typically yield only single-digit percentage improvements by comparison, making speculative decoding the higher-leverage inference optimization.; **GPU Snapshotting for Elastic Inference:** Modal integrates GPU state snapshotting directly into its autoscaling layer, capturing compiled Torch model state so subsequent cold starts resume significantly faster. This enables true scale-to-zero economics for custom model inference — serving companies like Suno and Runway — while handling diurnal and launch-spike traffic patterns across 17 cloud providers without dedicated data centers.

What did Akshat Bubna discuss on Latent Space?

Modal CTO Akshat Bubna explains how the company evolved from a serverless compute runtime built to replace Kubernetes into an AI infrastructure platform spanning elastic GPU inference, distributed training, and code-execution sandboxes — and why the same developer experience principles now apply to building for autonomous agents. Key topics include: **Agent Experience over Developer Experience:** Modal reoriented its SDK team from developer experience to agent experience, finding that co-locating infrastructure configuration inside code decorators — rather than sprawling YAML files — lets AI agents provision and modify compute environments faster. Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows.; **Speculative Decoding via Block-Based Drafting:** Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains with zero quality degradation. Kernel-level optimizations typically yield only single-digit percentage improvements by comparison, making speculative decoding the higher-leverage inference optimization..

How long is this episode of Latent Space?

This episode is 57 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Latent Space

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

July 8, 2026

57 min episode · 2 min read

Akshat Bubna

Episode

57 min

Read time

2 min

Topics

Career Growth, Remote Work, Startups

AI-Generated Summary

Published Jul 9, 2026

Key Takeaways

✓Agent Experience over Developer Experience: Modal reoriented its SDK team from developer experience to agent experience, finding that co-locating infrastructure configuration inside code decorators — rather than sprawling YAML files — lets AI agents provision and modify compute environments faster. Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows.
✓Speculative Decoding via Block-Based Drafting: Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains with zero quality degradation. Kernel-level optimizations typically yield only single-digit percentage improvements by comparison, making speculative decoding the higher-leverage inference optimization.
✓GPU Snapshotting for Elastic Inference: Modal integrates GPU state snapshotting directly into its autoscaling layer, capturing compiled Torch model state so subsequent cold starts resume significantly faster. This enables true scale-to-zero economics for custom model inference — serving companies like Suno and Runway — while handling diurnal and launch-spike traffic patterns across 17 cloud providers without dedicated data centers.
✓Sandbox Sidecars and IPv6 Overlay Networking: Modal sandboxes now support sidecar containers within a single pod, enabling Docker Compose-style multi-container workloads. An eBPF-enforced IPv6 overlay network allows Modal containers within the same workspace to address each other privately without encryption overhead, a primitive originally built for serverless distributed training that users independently discovered and repurposed for agent-to-agent communication.
✓Batch Tier Pricing for Non-Latency-Sensitive Workloads: Modal is building a batch pricing tier where users who accept up to 24-hour result windows receive substantially lower GPU costs. Demand comes primarily from computational biology and synthetic data generation workloads rather than LLM inference. Controlling the full scheduling stack across fungible GPU capacity in multiple regions makes this economically viable without dedicated hardware reservations.

What It Covers

Modal CTO Akshat Bubna explains how the company evolved from a serverless compute runtime built to replace Kubernetes into an AI infrastructure platform spanning elastic GPU inference, distributed training, and code-execution sandboxes — and why the same developer experience principles now apply to building for autonomous agents.

Key Questions Answered

•Agent Experience over Developer Experience: Modal reoriented its SDK team from developer experience to agent experience, finding that co-locating infrastructure configuration inside code decorators — rather than sprawling YAML files — lets AI agents provision and modify compute environments faster. Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows.
•Speculative Decoding via Block-Based Drafting: Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains with zero quality degradation. Kernel-level optimizations typically yield only single-digit percentage improvements by comparison, making speculative decoding the higher-leverage inference optimization.
•GPU Snapshotting for Elastic Inference: Modal integrates GPU state snapshotting directly into its autoscaling layer, capturing compiled Torch model state so subsequent cold starts resume significantly faster. This enables true scale-to-zero economics for custom model inference — serving companies like Suno and Runway — while handling diurnal and launch-spike traffic patterns across 17 cloud providers without dedicated data centers.
•Sandbox Sidecars and IPv6 Overlay Networking: Modal sandboxes now support sidecar containers within a single pod, enabling Docker Compose-style multi-container workloads. An eBPF-enforced IPv6 overlay network allows Modal containers within the same workspace to address each other privately without encryption overhead, a primitive originally built for serverless distributed training that users independently discovered and repurposed for agent-to-agent communication.
•Batch Tier Pricing for Non-Latency-Sensitive Workloads: Modal is building a batch pricing tier where users who accept up to 24-hour result windows receive substantially lower GPU costs. Demand comes primarily from computational biology and synthetic data generation workloads rather than LLM inference. Controlling the full scheduling stack across fungible GPU capacity in multiple regions makes this economically viable without dedicated hardware reservations.

Notable Moment

Modal built and shipped code-execution sandboxes in May 2023 — before the broader market recognized agent infrastructure as a category. The first published example placed an existing coding tool inside a self-iterating loop, effectively prototyping what became the standard agentic coding pattern roughly two years before widespread adoption.

Know someone who'd find this useful?

Episode Transcript

We're here with Akshat of Moto, CTO of Moto, together with Vivo. Congrats on your series c. Thank you. Your party yesterday was amazing. Yeah. All the photos and all the swag. We had a bunch of art installations, which, it was kind of fun seeing like our products on pedestals next to like Rodin. Very nice, very nice. When you started, it was not the GPU inference company, I mean, maybe it was in your mind? Take us back to the origin story. I actually first met Eric, who's the CEO through an investor, and back then Eric was already thinking about building, a new kind of runtime. And he he got there thinking through why are workflow orchestration products so hard to use? It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness, and custom images, and it's a terrible developer experience. And I'll inject for listeners who are new, we interviewed Eric two years ago, and there's a bit more of the story there from Spotify and all those things, And I actually came across Eric through a data council, because he he did that talk on the the sort of serverless container stack that that you guys did, which is like, that was my first, like, okay, I need to take models very seriously moment. But it's still very unclear, like, do I actually need all this for just my data pipelines? Yeah, I mean initially what we were thinking about was if we build a better runtime, it's a very useful primitive in itself, it's, there's a lot of things that, get solved by serverless functions, like you can do, ETL stuff, you can do job queues, you can do all this like bursty processing, which it turns out every company had needs for. But then we also were thinking about this as like, this is a primitive that we can build a whole collection of products on, which are very verticalized, so perhaps data engineering would have been the first one, but we were thinking about inference, back then it was more classical inference, like computer vision stuff and running XGBoost and whatnot. But we added GPUs to the product a year before JAD GPT came out, we just didn't think it would be that big of a deal. Yeah, just like add A100. Was there any like early key problem that really sparked off why you built it? Yeah primarily it's just, none of the tooling that was out there was built for, one a really great developer experience, and also there's a general trend of, a lot of the workloads that we were seeing were very, this is, I wish there's a better word for it, but compute heavy. Like they need, one like need a lot more resources, you need to burst up and down a lot, versus like Kubernetes designed for like slow scaling and, more for like web server use …

Get the full transcript (10,599 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Latent Space transcripts →

You just read a 3-minute summary of a 54-minute episode.

Get Latent Space summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Simulation: the new Scaling Law — Joon Sung Park, Simile AI

Aug 21 · 69 min

20VC (20 Minute VC)

20VC: The Best AI Companies Have Unique Data Acquisition Strategies | Will Simile Kill Kalshi, Polymarkets and NASDAQ | How to Sign Fortune 500 Companies As Customers in Weeks with Joon Sung Park, Simile

Aug 1

🔬The BioAI Phase Shift - Matthew McPartlon & Neil Patil, Chai Discovery

Aug 11 · 95 min

Masters of Scale

Pioneers of AI: John Deere's AI vision for future farms

Jul 11

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links. As an Amazon Associate, SignalCast earns from qualifying purchases.

Tools

ModalBy guest
by Modal
“Modal CTO Akshat Bubna explains how the company evolved from a serverless compute runtime built to replace Kubernetes into an AI infrastructure platform spanning elastic GPU inference, distributed training, and code-execution sandboxes”
DFlashBy guest
by Modal
“Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains”
Kubernetes
“Modal CTO Akshat Bubna explains how the company evolved from a serverless compute runtime built to replace Kubernetes”
Torch
“Modal integrates GPU state snapshotting directly into its autoscaling layer, capturing compiled Torch model state”
Docker Compose
“Modal sandboxes now support sidecar containers within a single pod, enabling Docker Compose-style multi-container workloads”

Products

Claude 4
by Anthropic
“Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows”
Amazon