What are the key takeaways from this Software Engineering Daily episode?

Key insights include: **CALM Theorem vs. CAP Theorem:** Where CAP frames consistency as a binary trade-off, CALM identifies a precise boundary: monotone programs — those where larger inputs produce strictly larger outputs without retractions — can achieve both consistency and partition availability simultaneously. Hydro uses this at compile time to label each network endpoint with its specific consistency guarantee, so engineers know exactly what guarantees they have.; **Distributed Safety via Type System:** Hydro encodes location into Rust's type system, making cross-node comparisons a compile error. Sending data between nodes requires explicit `.send()` syntax. Unordered message streams from clusters cannot be connected to order-sensitive operators without a `nondet` keyword — mirroring Rust's `unsafe` — preventing race conditions from compiling at all.; **Simulation Testing Scoped to Nondeterminism:** Hydro's built-in simulator only exhaustively tests code marked `nondet`, running all possible message orderings against developer-defined invariants on a local machine. Monotone, type-safe code paths are skipped entirely, making test suites faster and more targeted than conventional distributed system integration tests.

What did Joe Hellerstein discuss on Software Engineering Daily?

Joe Hellerstein, 30-year Berkeley distributed systems researcher now at AWS, explains how Hydro — a Rust framework — applies database query principles to general-purpose distributed programming, eliminating race conditions and ordering bugs through compile-time type checking and the CALM theorem. Key topics include: **CALM Theorem vs. CAP Theorem:** Where CAP frames consistency as a binary trade-off, CALM identifies a precise boundary: monotone programs — those where larger inputs produce strictly larger outputs without retractions — can achieve both consistency and partition availability simultaneously. Hydro uses this at compile time to label each network endpoint with its specific consistency guarantee, so engineers know exactly what guarantees they have.; **Distributed Safety via Type System:** Hydro encodes location into Rust's type system, making cross-node comparisons a compile error. Sending data between nodes requires explicit `.send()` syntax. Unordered message streams from clusters cannot be connected to order-sensitive operators without a `nondet` keyword — mirroring Rust's `unsafe` — preventing race conditions from compiling at all..

How long is this episode of Software Engineering Daily?

This episode is 50 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Software Engineering Daily

A Rust Framework to Simplify Distributed Systems

September 10, 2026

50 min episode · 2 min read

Joe Hellerstein

Episode

50 min

Read time

2 min

Topics

Remote Work, Artificial Intelligence, Software Development

AI-Generated Summary

Published Sep 10, 2026

Key Takeaways

✓CALM Theorem vs. CAP Theorem: Where CAP frames consistency as a binary trade-off, CALM identifies a precise boundary: monotone programs — those where larger inputs produce strictly larger outputs without retractions — can achieve both consistency and partition availability simultaneously. Hydro uses this at compile time to label each network endpoint with its specific consistency guarantee, so engineers know exactly what guarantees they have.
✓Distributed Safety via Type System: Hydro encodes location into Rust's type system, making cross-node comparisons a compile error. Sending data between nodes requires explicit `.send()` syntax. Unordered message streams from clusters cannot be connected to order-sensitive operators without a `nondet` keyword — mirroring Rust's `unsafe` — preventing race conditions from compiling at all.
✓Simulation Testing Scoped to Nondeterminism: Hydro's built-in simulator only exhaustively tests code marked `nondet`, running all possible message orderings against developer-defined invariants on a local machine. Monotone, type-safe code paths are skipped entirely, making test suites faster and more targeted than conventional distributed system integration tests.
✓Incremental Deployment Testing: Hydro's simulator can run multiple concurrent versions of a program simultaneously, modeling real-world incremental rollouts where v1 and v1.1 nodes communicate. This allows engineers to validate cross-version compatibility before launching staged deployments — a capability Hellerstein says only became apparent after working on production AWS systems.
✓Agentic Code Needs Compiler Guardrails: AI agents currently produce distributed systems bugs at high rates and lack reliable instincts for what tests to write. Using Hydro with Rust means the compiler enforces correctness properties the agent cannot reason about independently. Hellerstein frames this as the distributed equivalent of why Rust outperforms Python for agent-generated code.

What It Covers

Joe Hellerstein, 30-year Berkeley distributed systems researcher now at AWS, explains how Hydro — a Rust framework — applies database query principles to general-purpose distributed programming, eliminating race conditions and ordering bugs through compile-time type checking and the CALM theorem.

Key Questions Answered

•CALM Theorem vs. CAP Theorem: Where CAP frames consistency as a binary trade-off, CALM identifies a precise boundary: monotone programs — those where larger inputs produce strictly larger outputs without retractions — can achieve both consistency and partition availability simultaneously. Hydro uses this at compile time to label each network endpoint with its specific consistency guarantee, so engineers know exactly what guarantees they have.
•Distributed Safety via Type System: Hydro encodes location into Rust's type system, making cross-node comparisons a compile error. Sending data between nodes requires explicit `.send()` syntax. Unordered message streams from clusters cannot be connected to order-sensitive operators without a `nondet` keyword — mirroring Rust's `unsafe` — preventing race conditions from compiling at all.
•Simulation Testing Scoped to Nondeterminism: Hydro's built-in simulator only exhaustively tests code marked `nondet`, running all possible message orderings against developer-defined invariants on a local machine. Monotone, type-safe code paths are skipped entirely, making test suites faster and more targeted than conventional distributed system integration tests.
•Incremental Deployment Testing: Hydro's simulator can run multiple concurrent versions of a program simultaneously, modeling real-world incremental rollouts where v1 and v1.1 nodes communicate. This allows engineers to validate cross-version compatibility before launching staged deployments — a capability Hellerstein says only became apparent after working on production AWS systems.
•Agentic Code Needs Compiler Guardrails: AI agents currently produce distributed systems bugs at high rates and lack reliable instincts for what tests to write. Using Hydro with Rust means the compiler enforces correctness properties the agent cannot reason about independently. Hellerstein frames this as the distributed equivalent of why Rust outperforms Python for agent-generated code.

Notable Moment

Hellerstein reveals that his PhD adviser Michael Stonebreaker — who won the Turing Award and created Postgres — could not write software and avoided LaTeX entirely, yet was correct about architectural decisions consistently. Hellerstein argues this validates intuition-driven, detail-deferred thinking as a legitimate and high-value mode in computer science.

Know someone who'd find this useful?

Episode Transcript

Building software that runs across many machines is notoriously difficult. Developers have to grapple with problems such as race conditions, partial failures, and message ordering. Notably, one category of distributed software has largely escaped these burdens. A distributed database can spread a single query across thousands of machines, handling the coordination, failure, recovery, and ordering internally. This raises a natural question of why general purpose distributed programming can't feel the same way. This is a highly practical problem at AWS because the reliability of cloud infrastructure depends on getting distributed systems right at massive scale. Joe Hellerstein spent thirty years as a database and distributed systems researcher at UC Berkeley, where he pioneered much of the foundational thinking on applying database ideas to distributed programming. He is now at AWS, where he works to bring his research into production through Hydro, which is a Rust framework to bring declarative queries to general purpose distributed programming. In this episode, Joe joins Sean Falconer to discuss how ideas from the database world could make distributed programming dramatically simpler and safer. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. Joe, welcome to the show. Good to be here. Yeah, thanks for doing this. I'm excited about it. You have very interesting background. I don't think that we'll even be able to cover everything certainly that you've done in your career, but I think we have some interesting things to jump into. You spent thirty years at Berkeley working on databases, distributed systems, now you're at AWS. I guess, like, how would you describe this thread that has connected your research maybe going back to the nineties all the way to what you're working on today, which is Hydro, which we'll start to talk to about later? Yeah. Well, I think just to keep it focused because you work on a lot of things over thirty years, I'll pick the through thread like you said. I was trained up as a database person back in the nineties. Actually, right out of college, I went to IBM research in the very early nineties and got baptized in the database religion by the IBMers and then by Mike Stonebreaker at Berkeley, was my master's adviser, then by the Wisconsin team, which was the database powerhouse at the time, Dave DeWitt and others. So that was my training. One of the things that was cool about databases was that you could write a query, and it would run on any hardware you liked. So today, you can run an SQL query, and it'll run on an embedded device like your phone, or it'll run-in the cloud across thousands of machines, and it's the same query. And something that really bothered me as a software person was that programming wasn't like that at all. Programming computers was somehow different, and every time you got a new piece of hardware, you had to start over. …

Get the full transcript (10,504 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →