Grafana’s Approach to AI-Native Observability
Episode
50 min
Read time
2 min
Topics
Relationships, Startups, Leadership
AI-Generated Summary
Key Takeaways
- ✓Open Source as AI Training Advantage: Grafana's 12-year open source strategy created a massive public corpus of tutorials, GitHub repos, and documentation that foundation models from Anthropic, OpenAI, and Google trained on. This allowed a two-person team to build a functional AI assistant MVP in a single quarterly hackathon, bypassing expensive custom model training entirely.
- ✓Telemetry Volume as a Liability: The "measure all the things" philosophy from early observability culture now overwhelms engineering teams during incidents. Rather than collecting maximum data, teams should curate telemetry around service level objectives — specifically user-facing metrics like error rates and latency — rather than infrastructure metrics like CPU usage that don't directly reflect customer experience.
- ✓Knowledge Graphs for AI-Ready Observability: Grafana builds dynamic entity relationship graphs from live telemetry, mapping applications to pods, nodes, clusters, and regions in real time. Feeding this structured context to LLMs enables automated root cause analysis — for example, tracing a latency spike directly to a noisy neighbor consuming node resources — without requiring static CMDB databases that go stale immediately.
- ✓Agentic Deployment Requires Hard Gates, Not Trust: Rather than trusting AI agents to self-govern deployments, Grafana embeds hard CI/CD blocks that prevent globally scoped changes, flag 10x cost increases, and enforce small blast radius deployments. These same gates that protect against human error also constrain agents, making limited autonomous deployment — like auto-generating rollback PRs — operationally viable today.
- ✓AI Observability Requires Supplemental Data Stores: Standard OpenTelemetry traces cannot contain the full conversation history generated by LLM-based agents, as conversation volumes exceed single span limits. Teams building AI features in production should architect supplemental SQL-style stores for conversation data, then use a visualization layer like Grafana to correlate that data with traditional APM traces in one unified view.
What It Covers
Grafana Labs co-founder Anthony Woods discusses how AI-generated code and autonomous agents are reshaping software observability, why telemetry data volume has become a liability, how Grafana is rebuilding its tools for agent-first consumption, and what governance gaps in agentic systems pose the greatest operational risks.
Key Questions Answered
- •Open Source as AI Training Advantage: Grafana's 12-year open source strategy created a massive public corpus of tutorials, GitHub repos, and documentation that foundation models from Anthropic, OpenAI, and Google trained on. This allowed a two-person team to build a functional AI assistant MVP in a single quarterly hackathon, bypassing expensive custom model training entirely.
- •Telemetry Volume as a Liability: The "measure all the things" philosophy from early observability culture now overwhelms engineering teams during incidents. Rather than collecting maximum data, teams should curate telemetry around service level objectives — specifically user-facing metrics like error rates and latency — rather than infrastructure metrics like CPU usage that don't directly reflect customer experience.
- •Knowledge Graphs for AI-Ready Observability: Grafana builds dynamic entity relationship graphs from live telemetry, mapping applications to pods, nodes, clusters, and regions in real time. Feeding this structured context to LLMs enables automated root cause analysis — for example, tracing a latency spike directly to a noisy neighbor consuming node resources — without requiring static CMDB databases that go stale immediately.
- •Agentic Deployment Requires Hard Gates, Not Trust: Rather than trusting AI agents to self-govern deployments, Grafana embeds hard CI/CD blocks that prevent globally scoped changes, flag 10x cost increases, and enforce small blast radius deployments. These same gates that protect against human error also constrain agents, making limited autonomous deployment — like auto-generating rollback PRs — operationally viable today.
- •AI Observability Requires Supplemental Data Stores: Standard OpenTelemetry traces cannot contain the full conversation history generated by LLM-based agents, as conversation volumes exceed single span limits. Teams building AI features in production should architect supplemental SQL-style stores for conversation data, then use a visualization layer like Grafana to correlate that data with traditional APM traces in one unified view.
Notable Moment
Woods describes a scenario that concerns him most: agent-to-agent communication across organizational boundaries, where two companies' autonomous systems interact and cause harm. When something goes wrong in that chain, no clear accountability framework exists yet to determine who is responsible for remediation.
Episode Transcript
Advanced software systems have long been more complex than any single engineer can fully understand. Observability is the established solution to this problem. But with AI agents now generating code, deploying changes, and operating autonomously, the challenge of understanding large software systems is entering a new dimension. Grafana is an open source observability platform and one of the most widely used in the world. The company builds tools that help teams collect, visualize, and act on telemetry data across logs, metrics, and traces. They are now extending that capability into the agentic era with AI powered investigation and monitoring tools. Anthony Woods is a cofounder of Grafana Labs. In this episode, he joins Matt Merrill to discuss how AI generated code is straining software operations, why telemetry data volume has become as much a problem as a solution, how Grafana is adapting to a world where agents are the primary consumers of observability data, and what keeps him up at night about where the industry is headed. Matt Merrill is a software engineering leader with over twenty years of experience building and scaling software teams across enterprise and product focused organizations. His background is in back end development, cloud architecture, and distributed systems design. He currently architects and delivers software products and leads a team of engineers at Dept Agency. You can learn more about his work at co.the other mattm.com. Hello. Welcome to Software Engineering Daily. I am Matt Merrill, and I am here today with Anthony Woods, the founder of Grafana. Before we start, I am going to let him introduce himself. Thanks, Matt. Thank you for having me on the show. Yeah. So I'm Anthony. I'm one of the co founders, there were three of us, at Grafana Labs. I'm based in sunny Perth in Western Australia. It's a beautiful city. It's just far away from everything, which is okay. I spent a lot of time on a plane traveling around. Yeah. I have a background in tech. I started off as a systems and network engineer then, you know, kept building automation tooling until one day I found I was just writing code full time and become a software developer. And so that was a lot of fun, you know, being able to bring together all different parts of the system, you know, with tools we can build. And about twelve years ago, Raj, Talkal and I, we started a little company called Grafana Labs, right, based on, Grafana, the open source project that Talkal had created. And we took that love of open source and of helping people, you know, understand the data and the telemetry that are getting out of systems and turned it into a business. Pat is awesome. I am super excited to talk to you because I am a back end nerd. I've done pretty much back end stuff my whole life. I've always been at the intersection of back end and DevOps, and so super excited. So today, I want …
Get the full transcript (11,611 words) + summary by email — free
One-time email with the complete transcript and AI summary of this episode. No account needed.
One email, no spam. We’ll also show you what SignalCast does.
You just read a 3-minute summary of a 47-minute episode.
Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.
Pick Your Podcasts — FreeKeep Reading
More from Software Engineering Daily
Rebuilding the Cloud for AI Agent Code
Aug 13 · 49 min
How I AI
How I run autonomous coding agents from my phone with OpenAI Symphony + Linear | Alessio Fanelli (Kernel Labs)
Jul 6
More from Software Engineering Daily
SED News: The Kimi Moment, Runaway AI, and Tokenmaxxing
Aug 11 · 48 min
Odd Lots
This Is How to Tell if Writing Was Made by AI
Apr 2
Books, tools, and gear mentioned in this episode
SignalCast may earn commission on purchases via these links.
Tools
- GrafanaBy guest
by Grafana Labs
“Grafana Labs co-founder Anthony Woods discusses how AI-generated code and autonomous agents are reshaping software observability... Grafana is rebuilding its tools for agent-first consumption”
“Standard OpenTelemetry traces cannot contain the full conversation history generated by LLM-based agents, as conversation volumes exceed single span limits.”
More from Software Engineering Daily
We summarize every new episode. Want them in your inbox?
Rebuilding the Cloud for AI Agent Code
SED News: The Kimi Moment, Runaway AI, and Tokenmaxxing
The Terminal as an Agentic Interface
AI-Powered Threats to the Software Supply Chain
Docker and Sandboxing AI Agents
Similar Episodes
Related episodes from other podcasts
How I AI
Jul 6
How I run autonomous coding agents from my phone with OpenAI Symphony + Linear | Alessio Fanelli (Kernel Labs)
Odd Lots
Apr 2
This Is How to Tell if Writing Was Made by AI
Hard Fork
Mar 27
The Ezra Klein Show: How Fast Will A.I. Agents Rip Through the Economy?
No Priors: Artificial Intelligence | Technology | Startups
Mar 12
From Coder to Manager: Navigating the Shift to Agentic Engineering with Notion Co-Founder Simon Last
Techmeme Ride Home
Mar 10
Meta Plumps For Bot Social Networks
Explore Related Topics
This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.
Read this week's Startups & Product Podcast Insights — cross-podcast analysis updated weekly.
You're clearly into Software Engineering Daily.
Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.
Start My Monday DigestNo credit card · Unsubscribe anytime