Is Claude Mythos “Terrifying”? | AI Reality Check

What It Covers

Cal Newport analyzes whether Claude Mythos, Anthropic's newest AI model, represents a genuine cybersecurity breakthrough. Using independent security researcher findings and UK AI Security Institute benchmark data, Newport argues the model's capabilities show incremental improvement over existing models, not the paradigm-shifting threat Anthropic's marketing campaign suggested.

Key Questions Answered

• •LLM cybersecurity baseline: Security researchers have used LLMs to exploit vulnerabilities since GPT-4, which successfully exploited 87% of presented vulnerabilities in a 2024 IBM study. Anthropic's own earlier Opus 4.6 model already identified over 500 exploitable zero-day vulnerabilities. Mythos did not introduce a new capability category — it continues a three-to-four-year-old trend.
• •Independent replication test: Researchers from Hugging Face tested the specific vulnerabilities Anthropic highlighted in the Mythos announcement against small, cheap open-weight models. Eight out of eight models — including one with only 3.6 billion parameters costing 11 cents per million tokens — detected the same flagship FreeBSD exploit Anthropic used as its headline example.
• •AISI benchmark results: The UK AI Security Institute tested Mythos directly on capture-the-flag security tasks. Performance clustered near GPT-5 and Opus 4.6, with no disproportionate jump. On a contrived 32-step attack scenario, Mythos completed 22 steps on average versus Opus 4.6's 16 — a measurable but incremental gain, not a capability threshold crossing.
• •Agent tuning vs. model intelligence: Improvements in LLM exploitation benchmarks may reflect better agent compatibility rather than deeper cybersecurity understanding. Because models require external agents to execute multi-step attacks, recent performance gains could stem from companies tuning models to follow longer instruction chains for coding agents — a separate commercial priority unrelated to security reasoning.
• •Marketing vs. capability gap: When evaluating AI announcements, cross-reference company claims against independent researcher replication tests before drawing conclusions. Anthropic briefed government officials and journalists directly, generating Thomas Friedman-level alarm. Previous model releases showing comparable benchmark jumps received no equivalent coverage, revealing that narrative framing — not capability magnitude — drove the reaction.