Snap’s Secret to Processing 10 Petabytes a Day: GPU-Accelerated Spark | NVIDIA AI Podcast Ep. 298

We were able to cut almost about 76% of our job costs as a result of this migration. Seventy six? Seventy six. Wow. It's it's phenomenal. Yeah. Right? I mean, for for the engineers out there, like, we were able to cut down the number of course required by, like, 62%. Amazing. The the memory footprint print, we could we could drop it by, like, 80%. So phenomenal results. The results speak for themselves. Welcome to the NVIDIA AI podcast. I'm Noah Kravitz. I'm here with Pruevi Vatala. Pruevi is the head of engineering platforms at Snap, and we're here to talk about data processing and in particular, how a social platform with more than 940,000,000 active users accelerated their data pipeline. Prue, welcome to the NVIDIA AI podcast. Thanks so much for taking the time to join us. Yeah. Thanks for having me here, Noah. So maybe we can start with the basics. Tell us a little bit about well, about what Snap is now. I I'm old, but I still think of it, you know, the Snap Glasses and everything, but Snapchat, obviously, a huge social platform. So maybe tell us a little bit about Snap and then your role there. Absolutely. Yeah. I mean, Snapchat at this point is pretty much a household name. You know? It's, Snap as a company, it's it's interesting that you bring up the spectacles because Snap as a company believes that camera is at the center of, you know, improving how people communicate and improve their lives, you know, in the digital world, so to speak. So we've been steadfast on that belief and, you know, Snap right now is a is is at the, intersection of, augmented reality, AI, and visual communication. Like you said, serving close to a billion monthly active users. I've been at Snap for a while now, and, I lead a multifaceted organization. We do, a little bit of it has to do with big data infrastructure, a little bit of it with developer productivity, and a little bit of it, with enterprise AI, and whatnot. So yeah. And so when we talk about accelerating data processing, what does that mean to you? What does that mean for Snap? And thinking about the scale that you operate on, just talk a little bit about what it means to accelerate data at that level. Absolutely. That's a great question. Like, as you can imagine with as many users as we have and snap Snapchat in particular is a very complex application. So you can imagine the scale at which we operate, especially on the data processing side. We are dealing with my team's experimentation platform is dealing with 10 plus petabytes each day. It's a massive scale. Right? It's a huge scale. Yeah. And then we have a strict SLA in the morning because experimentation results need to be ready for developers, product managers, data scientists to act on as early as possible, so that, you know, …