Are We About to Lose Control of AI? | AI Reality Check

What It Covers

Cal Newport analyzes Anthropic's "When AI Builds Itself" report, which warns of recursive self-improvement leading to loss of human control. Newport examines the three core charts cited as evidence and argues the data reflects coding tool maturation, not an imminent AI autonomy crisis.

Key Questions Answered

• •Recursive Self-Improvement Misread: Anthropic's report shows Claude Code session success rates on open-ended problems rising from roughly 20% to 70%, but this jump reflects the introduction of mature coding harnesses in fall 2025, not AI becoming self-aware. The baseline starts at zero because the tools simply didn't exist before that date.
• •AI Breakthroughs Require Ideas, Not Faster Code: The three advances behind modern generative AI — Hinton's backpropagation, Google's attention transformer architecture, and Kaplan's scaling laws at OpenAI — were scientific insights, not engineering outputs. Speeding up programmer productivity via LLM tools does not accelerate the discovery of the next foundational AI breakthrough.
• •Coding Harnesses Are Fully Deterministic: AI software development tools combine an LLM with a human-written coding harness built from conditional logic, pattern matching, and hard-coded rules. All actions and tool access run through that harness. To restrict any capability with 100% certainty, simply remove it from the harness — no ambiguity exists.
• •More Apps, Less Usage: A Financial Times chart tracking iOS app releases after AI coding tools launched in 2025 shows app volume rising sharply while apps with significant user engagement held flat or declined. AI accelerates output quantity but does not automatically generate economically useful or widely adopted products.
• •Anthropic's Slowdown Caveat Undermines the Warning: The report's apparent call for a global AI development pause contains a built-in escape clause — Anthropic states it would only slow down if all actors worldwide did simultaneously. Otherwise, it continues at full speed. This framing offers no actionable safety mechanism and functions more as public positioning than a concrete proposal.