Anthony Woods

AI Summary

→ WHAT IT COVERS Grafana Labs co-founder Anthony Woods discusses how AI-generated code and autonomous agents are reshaping software observability, why telemetry data volume has become a liability, how Grafana is rebuilding its tools for agent-first consumption, and what governance gaps in agentic systems pose the greatest operational risks. → KEY INSIGHTS - **Open Source as AI Training Advantage:** Grafana's 12-year open source strategy created a massive public corpus of tutorials, GitHub repos, and documentation that foundation models from Anthropic, OpenAI, and Google trained on. This allowed a two-person team to build a functional AI assistant MVP in a single quarterly hackathon, bypassing expensive custom model training entirely. - **Telemetry Volume as a Liability:** The "measure all the things" philosophy from early observability culture now overwhelms engineering teams during incidents. Rather than collecting maximum data, teams should curate telemetry around service level objectives — specifically user-facing metrics like error rates and latency — rather than infrastructure metrics like CPU usage that don't directly reflect customer experience. - **Knowledge Graphs for AI-Ready Observability:** Grafana builds dynamic entity relationship graphs from live telemetry, mapping applications to pods, nodes, clusters, and regions in real time. Feeding this structured context to LLMs enables automated root cause analysis — for example, tracing a latency spike directly to a noisy neighbor consuming node resources — without requiring static CMDB databases that go stale immediately. - **Agentic Deployment Requires Hard Gates, Not Trust:** Rather than trusting AI agents to self-govern deployments, Grafana embeds hard CI/CD blocks that prevent globally scoped changes, flag 10x cost increases, and enforce small blast radius deployments. These same gates that protect against human error also constrain agents, making limited autonomous deployment — like auto-generating rollback PRs — operationally viable today. - **AI Observability Requires Supplemental Data Stores:** Standard OpenTelemetry traces cannot contain the full conversation history generated by LLM-based agents, as conversation volumes exceed single span limits. Teams building AI features in production should architect supplemental SQL-style stores for conversation data, then use a visualization layer like Grafana to correlate that data with traditional APM traces in one unified view. → NOTABLE MOMENT Woods describes a scenario that concerns him most: agent-to-agent communication across organizational boundaries, where two companies' autonomous systems interact and cause harm. When something goes wrong in that chain, no clear accountability framework exists yet to determine who is responsible for remediation. 💼 SPONSORS [{"name": "Vision Agents by Stream", "url": "https://visionagents.ai"}, {"name": "GuardSquare", "url": "https://www.guardsquare.com"}] 🏷️ AI Observability, OpenTelemetry, Agentic Systems, Site Reliability Engineering, Grafana Labs