Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

What It Covers

Modal CTO Akshat Bubna explains how the company evolved from a serverless compute runtime built to replace Kubernetes into an AI infrastructure platform spanning elastic GPU inference, distributed training, and code-execution sandboxes — and why the same developer experience principles now apply to building for autonomous agents.

Key Questions Answered

• •Agent Experience over Developer Experience: Modal reoriented its SDK team from developer experience to agent experience, finding that co-locating infrastructure configuration inside code decorators — rather than sprawling YAML files — lets AI agents provision and modify compute environments faster. Agents one-shot Modal code reliably post-Claude 4, making decorator-based infrastructure a practical default for agentic workflows.
• •Speculative Decoding via Block-Based Drafting: Modal's open-source DFlash uses block-based speculative decoding — predicting multiple tokens simultaneously with a smaller draft model, then batch-verifying with the larger model — to achieve two-to-four times throughput gains with zero quality degradation. Kernel-level optimizations typically yield only single-digit percentage improvements by comparison, making speculative decoding the higher-leverage inference optimization.
• •GPU Snapshotting for Elastic Inference: Modal integrates GPU state snapshotting directly into its autoscaling layer, capturing compiled Torch model state so subsequent cold starts resume significantly faster. This enables true scale-to-zero economics for custom model inference — serving companies like Suno and Runway — while handling diurnal and launch-spike traffic patterns across 17 cloud providers without dedicated data centers.
• •Sandbox Sidecars and IPv6 Overlay Networking: Modal sandboxes now support sidecar containers within a single pod, enabling Docker Compose-style multi-container workloads. An eBPF-enforced IPv6 overlay network allows Modal containers within the same workspace to address each other privately without encryption overhead, a primitive originally built for serverless distributed training that users independently discovered and repurposed for agent-to-agent communication.
• •Batch Tier Pricing for Non-Latency-Sensitive Workloads: Modal is building a batch pricing tier where users who accept up to 24-hour result windows receive substantially lower GPU costs. Demand comes primarily from computational biology and synthetic data generation workloads rather than LLM inference. Controlling the full scheduling stack across fungible GPU capacity in multiple regions makes this economically viable without dedicated hardware reservations.