Skip to main content
Software Engineering Daily

New Relic and Agentic DevOps with Nic Benders

46 min episode · 2 min read
·
Nic Benders

Episode

46 min

Read time

2 min

Topics

Career Growth, Investing, Fundraising & VC

AI-Generated Summary

Key Takeaways

  • Observability Era Progression: New Relic identifies three distinct phases: instrumentation (pre-2013), data platform (NRDB launch, 2013–2014), and intelligence (current). A fourth "action era" is emerging where systems autonomously remediate issues before engineers are paged. Teams should evaluate whether their tooling strategy reflects this progression or remains anchored in dashboard-centric thinking.
  • LLM + Statistics Hybrid Architecture: Feeding raw petabyte-scale telemetry directly into LLMs is cost-prohibitive and ineffective. The practical architecture runs statistical anomaly detection first to reduce billions of data points to thousands of relevant signals, then passes those filtered results with temporal and spatial service-graph context into an LLM reasoning layer for root-cause synthesis.
  • Alert Fatigue Root Cause: Adding more alerts measurably delays incident response because engineers learn to wait and see if alerts self-resolve. The structural fix is not alert tuning but replacing alert configuration entirely with outcome-based observability: define the business signals that matter most, then let the intelligence layer determine when autonomous action, human escalation, or passive logging is appropriate.
  • AI Observability Golden Signals: Monitoring AI-powered applications requires tracking token consumption, response quality via sampled LLM-judge evaluation (e.g., routing one-in-a-thousand queries to a higher-capability model for scoring), cost per interaction, and sentiment drift. Quality degradation between model versions — such as moving from one Claude Sonnet release to another — surfaces through this sampling pattern before customer complaints appear.
  • Business Metric as Source of Truth: All technical observability signals — CPU, memory, error rates, AI response quality — are diagnostic. The authoritative signal is whether the application achieves its business objective, such as sales per minute or conversion completion. Teams should instrument and display this primary metric separately and treat all infrastructure alerting as subordinate diagnostic tooling.

What It Covers

New Relic Chief Technology Strategist Nic Benders traces observability's evolution through three eras — instrumentation, data platform, and intelligence — with host Lee Atchison, covering how LLMs combine with statistical tools to surface signals from massive datasets, how to monitor AI systems, and what agentic DevOps means for software engineering careers.

Key Questions Answered

  • Observability Era Progression: New Relic identifies three distinct phases: instrumentation (pre-2013), data platform (NRDB launch, 2013–2014), and intelligence (current). A fourth "action era" is emerging where systems autonomously remediate issues before engineers are paged. Teams should evaluate whether their tooling strategy reflects this progression or remains anchored in dashboard-centric thinking.
  • LLM + Statistics Hybrid Architecture: Feeding raw petabyte-scale telemetry directly into LLMs is cost-prohibitive and ineffective. The practical architecture runs statistical anomaly detection first to reduce billions of data points to thousands of relevant signals, then passes those filtered results with temporal and spatial service-graph context into an LLM reasoning layer for root-cause synthesis.
  • Alert Fatigue Root Cause: Adding more alerts measurably delays incident response because engineers learn to wait and see if alerts self-resolve. The structural fix is not alert tuning but replacing alert configuration entirely with outcome-based observability: define the business signals that matter most, then let the intelligence layer determine when autonomous action, human escalation, or passive logging is appropriate.
  • AI Observability Golden Signals: Monitoring AI-powered applications requires tracking token consumption, response quality via sampled LLM-judge evaluation (e.g., routing one-in-a-thousand queries to a higher-capability model for scoring), cost per interaction, and sentiment drift. Quality degradation between model versions — such as moving from one Claude Sonnet release to another — surfaces through this sampling pattern before customer complaints appear.
  • Business Metric as Source of Truth: All technical observability signals — CPU, memory, error rates, AI response quality — are diagnostic. The authoritative signal is whether the application achieves its business objective, such as sales per minute or conversion completion. Teams should instrument and display this primary metric separately and treat all infrastructure alerting as subordinate diagnostic tooling.

Notable Moment

Benders describes how every post-incident retrospective ends identically: teams resolve to add more alerts, which over years produces alert-for-everything environments that paradoxically slow response times. The actual fix, he argues, is eliminating the need for human-configured alerts altogether through autonomous remediation systems.

Know someone who'd find this useful?

Episode Transcript

Observability emerged from the need to understand complex software systems and involves tracking metrics, logs, and traces, so engineers can detect and diagnose problems before they affect users. However, modern applications often encompass hundreds of services, containers, and dependencies, generating more observability data than dashboards and alerts alone can effectively surface. New Relic is a leading observability with a history that spans the full arc of modern software operations. Today, they are working to apply AI to move observability beyond passive monitoring toward active intelligence, where systems can surface what matters, reduce alert noise, and ultimately take autonomous action before problems reach engineers or users. Nick Benders is the Chief Technology Strategist at New Relic, where he has worked for sixteen years. In this episode, Nick joins Lee Acheson to discuss the evolution of observability from dashboards and alerts to AI driven intelligence, how LLMs and statistical tools work together to surface meaningful signals from massive datasets, the emerging challenge of observing AI systems themselves, and what the rise of AI means for the future of software engineering as a profession. This episode is hosted by Lee Atchison. Lee Atchison is a software architect, author, and thought leader on cloud computing and application modernization. His best selling book, Architecting for Scale, is an essential resource for technical teams looking to maintain high availability and manage risk in their cloud environments. Lee is the host of his podcast, Modern Digital Business, produced for people looking to build and grow their digital business. Listen at mdb.fm. Follow Lee at softwarearchitectureinsights.com, and see all his content at leeatchison.com. AI and observability. How exactly do they work together? My guest today is Nick Benders. Nick is the chief technology strategist for New Relic, one of the major observability platforms that is now focused on AI. And Nick is also a personal friend of mine. So Nick, welcome to Software Engineering Daily. Thanks, Lee. It's great to be here. Now you and I go back a long time from New Relic days, early in the New Relic days. So matter of fact, we've just spent a little bit of time talking about some of those earlier days. But can you catch listeners up on what you've been doing since those early days in New Relic and what your role as chief technology strategist really involves? Absolutely. So when I think about New Relic's journey and really in some ways the industry's journey, back when we started at New Relic, we were very much solidly in this instrumentation era. Like, the thing that we sat down at our desks and tried to figure out every day was how can we instrument more of the systems that matter to people? Oh, well, we started with Ruby. How do we get Java? How do we get ..net? How do we get Python? How do we get into the browser or onto mobile apps or add a new library? And pretty soon, we were instrumented so many things that …

Get the full transcript (8,520 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Software Engineering Daily transcripts →

You just read a 3-minute summary of a 43-minute episode.

Get Software Engineering Daily summarized like this every Monday — plus up to 2 more podcasts, free.

Pick Your Podcasts — Free

Keep Reading

Books, tools, and gear mentioned in this episode

SignalCast may earn commission on purchases via these links.

Tools

  • by New Relic

    New Relic identifies three distinct phases: instrumentation (pre-2013), data platform (NRDB launch, 2013–2014), and intelligence (current)

company

  • New Relic Chief Technology Strategist Nic Benders traces observability's evolution through three eras — instrumentation, data platform, and intelligence

More from Software Engineering Daily

We summarize every new episode. Want them in your inbox?

Similar Episodes

Related episodes from other podcasts

Explore Related Topics

This podcast is featured in Best Cybersecurity Podcasts (2026) — ranked and reviewed with AI summaries.

Read this week's Investing & Markets Podcast Insights — cross-podcast analysis updated weekly.

You're clearly into Software Engineering Daily.

Every Monday, we deliver AI summaries of the latest episodes from Software Engineering Daily and 192+ other podcasts. Free for one show.

Start My Monday Digest

No credit card · Unsubscribe anytime