What are the key takeaways from this Software Engineering Daily episode?

Key insights include: **eBPF kernel programming:** Developers write C programs that compile to bytecode, pass through a safety verifier, then load into the Linux kernel without modules or restarts, enabling dynamic packet filtering and system tracing in production environments.; **Hardware offloading strategy:** ByteDance combines eBPF with smart NIC hardware by using a slow path-fast path separation where eBPF processes initial packets, then an agent translates rules to hardware that caches them for thirty-second intervals, bypassing kernel overhead.; **Container networking at scale:** Traditional Kubernetes service discovery becomes a bottleneck above 100,000 machines because indexing all backend containers creates unacceptable overhead, requiring ByteDance to build custom service discovery frameworks that operate without global state management.

What did Chen Tang discuss on Software Engineering Daily?

ByteDance engineer Chen Tang explains how the company uses eBPF technology to manage container networking across over one million servers, replacing traditional virtual switches with kernel-level packet routing for improved efficiency and scalability. Key topics include: **eBPF kernel programming:** Developers write C programs that compile to bytecode, pass through a safety verifier, then load into the Linux kernel without modules or restarts, enabling dynamic packet filtering and system tracing in production environments.; **Hardware offloading strategy:** ByteDance combines eBPF with smart NIC hardware by using a slow path-fast path separation where eBPF processes initial packets, then an agent translates rules to hardware that caches them for thirty-second intervals, bypassing kernel overhead..

How long is this episode of Software Engineering Daily?

This episode is 47 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Software Engineering Daily

ByteDance’s Container Networking Stack with Chen Tang

July 1, 2025

47 min episode · 2 min read

Chen Tang

Episode

47 min

Read time

2 min

Topics

Productivity, Leadership, Software Development

AI-Generated Summary

Published Dec 25, 2025

Key Takeaways

✓eBPF kernel programming: Developers write C programs that compile to bytecode, pass through a safety verifier, then load into the Linux kernel without modules or restarts, enabling dynamic packet filtering and system tracing in production environments.
✓Hardware offloading strategy: ByteDance combines eBPF with smart NIC hardware by using a slow path-fast path separation where eBPF processes initial packets, then an agent translates rules to hardware that caches them for thirty-second intervals, bypassing kernel overhead.
✓Container networking at scale: Traditional Kubernetes service discovery becomes a bottleneck above 100,000 machines because indexing all backend containers creates unacceptable overhead, requiring ByteDance to build custom service discovery frameworks that operate without global state management.
✓RDMA integration technique: eBPF enables RDMA direct memory access for containers by first identifying destination locations through kernel hooks, then passing connectivity information to NICs that can bypass kernel stack entirely for subsequent packets between containerized applications.

What It Covers

ByteDance engineer Chen Tang explains how the company uses eBPF technology to manage container networking across over one million servers, replacing traditional virtual switches with kernel-level packet routing for improved efficiency and scalability.

Key Questions Answered

•eBPF kernel programming: Developers write C programs that compile to bytecode, pass through a safety verifier, then load into the Linux kernel without modules or restarts, enabling dynamic packet filtering and system tracing in production environments.
•Hardware offloading strategy: ByteDance combines eBPF with smart NIC hardware by using a slow path-fast path separation where eBPF processes initial packets, then an agent translates rules to hardware that caches them for thirty-second intervals, bypassing kernel overhead.
•Container networking at scale: Traditional Kubernetes service discovery becomes a bottleneck above 100,000 machines because indexing all backend containers creates unacceptable overhead, requiring ByteDance to build custom service discovery frameworks that operate without global state management.
•RDMA integration technique: eBPF enables RDMA direct memory access for containers by first identifying destination locations through kernel hooks, then passing connectivity information to NICs that can bypass kernel stack entirely for subsequent packets between containerized applications.

Notable Moment

Chen reveals that ByteDance can inject observability code directly into the kernel of live production containers, collect diagnostic data from specific function calls and contexts, then remove the tracing program without any system restart or service interruption.

Know someone who'd find this useful?

Episode Transcript

ByteDance is a global technology company operating a wide range of content platforms around the world and is best known for creating TikTok. The company operates at a massive scale, which naturally presents challenges in ensuring performance and stability across its data centers. It has over a million servers running containerized applications, and this required the company to find a networking solution that could handle high throughput while maintaining stability. EBPF is a technology for dynamically and safely reprogramming the Linux kernel. ByteDance leveraged eBPF to successfully implement a decentralized networking solution that improved efficiency, scalability, and performance. Chen Tang is an engineer at ByteDance, where he worked on redesigning the company's container networking stack using eBPF. In this episode, Chen joins the show with Kevin Ball to talk about eBPF, the problems it solves, and how it was used at ByteDance. Kevin Ball or Kate Ball is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders. He cofounded and served as CTO for two companies, founded the San Diego JavaScript meetup, and organizes the AI in action discussion group through Latent Space. Check out the show notes to follow Keball on Twitter or LinkedIn, or visit his website, keball.llc. Jen, welcome to the show. Hi, Kevin. I'm excited to get to talk to you. Yeah. Me too. Let's maybe start with you can introduce yourself. Just tell me a little bit about your background and how you got involved doing cloud native stuff and networking. Okay. My name is Chen, and I'm currently a software engineer in Python. So my job is focusing on networking parts in our data center to keep, like, every service in our data centers especially deployed in containers rounding and to make sure their connectivity and stability. And in this field, we use a lot of different technologies. We, use in kernel technology and hardware technologies. But in the kernel part, we use eBPF. It's a customized program you wrote, and you can somehow to put it wrong inside the kernel without loading a kernel module. I think for us, it's the eBPF technology has been developed rapidly in the recent decades, and now it's very popular. You can using it in not just networking, but, basically, you can do everything with eBPF. And you can just write your program and you want them to run inside the kernel, and you don't have to be afraid that your code might jeopardize the entire system. I think that's worth digging into, especially for older developers like me. I remember to get anything into the kernel when I started, it was this months or years long process where you would go back and forth on email and all these different things. But eBPF, as I understand it, lets you essentially run sandboxed code similar to how you might run JavaScript in a browser. Is that a fair analogy? I'm not quite familiar with the JavaScript, but, yes, I think …

Get the full transcript (7,541 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →