AI researchers debate how close we are to recursive self-improvement

What It Covers

Three AI researchers — Baron Millich (Zyphra CTO), John Schulman (Thinking Machines chief scientist, ex-OpenAI), and Charlie O'Neil (Base Ten) — debate the technical barriers to recursive self-improvement, why RL has outperformed expectations, how distillation undermines frontier lab advantages, and what bottlenecks remain before AI systems can fully automate AI research and development.

Key Questions Answered

• •RSI Bottleneck — Objective Specification: The last human role in AI research will be defining what the system should optimize for, not executing the optimization itself. Specifying objectives for alignment, model behavior, and post-training rubrics resists automation because no verifiable signal exists to confirm correctness. Even with fully automated technical execution, humans must still decide what "good behavior" means across every deployment domain, making alignment specification the final durable human job in the AI development pipeline.
• •Distillation vs. Centralization: Frontier model advantages erode faster than expected because any capability learnable via RL can be distilled into smaller models using relatively few behavioral traces. Chinese labs reportedly purchase routing-service data — real user prompts from people accessing US frontier models via proxy — giving them near-optimal prompt distributions for distillation. This means prompt distribution quality, not raw model access, determines distillation success, and labs without deployment data can partially compensate by purchasing third-party usage datasets.
• •RL Signal-to-Noise Advantage: RL outperforms supervised fine-tuning not because it provides more information, but because it provides higher signal-to-noise ratio. SFT forces the model to match every reasoning token in a trace, drowning the correct-answer signal in noise. RL delivers only the outcome bit — right or wrong — so the gradient update is not diluted by irrelevant token-level imitation. Mid-training on synthetic reasoning data handles roughly 80% of capability gains; RL then fine-tunes the remaining policy with minimal but precise updates.
• •Horizon Generalization Over Domain Generalization: RL on math does not reliably transfer reasoning improvements to coding or other domains — horizontal generalization remains weak. What does generalize robustly is horizon generalization: models trained on progressively longer tasks learn to sustain productive computation across extended sequences, then apply that persistence to new domains. Research cited from Edgebench shows the maximum effective working horizon for frontier models doubles approximately every three months, which correlates strongly with benchmark performance improvements across diverse task categories.
• •Cumulative vs. Non-Stationary Task Split: RSI is likely easier to automate than general real-world deployment because AI research is a cumulative task — each discovery (attention, MoE, GRPO) permanently enters the training stack without needing rediscovery. Real-world tasks like legal work or client management are non-stationary: relationships, processes, and implicit knowledge shift continuously, requiring ongoing adaptation rather than accumulation. This distinction predicts that labs will achieve automated AI R&D before achieving robust continual learning for deployed enterprise agents.
• •Data Explains ~9x, Architecture ~3x of Compute Efficiency Gains: An investigation pairing all training recipes from 2019 to present with all datasets from 2019 to present found that data quality improvements account for roughly nine times the compute efficiency gains, while architectural improvements account for roughly three times — at small scale. Architecture changes matter less as multiplicative efficiency gains and more as qualitative unlocks: GQA enabling million-token contexts, for example, makes previously unusable long-context data accessible, indirectly amplifying data's contribution at scales where the architecture constraint is binding.
• •Continual Learning Breaks at Micro-Scale: Continual mid-training works at large scale because noise washes out across diverse data distributions, but degrades at the micro-scale of a single deployed model updating on narrow domain data. Iterative SFT on successful traces causes catastrophic forgetting of earlier capabilities; on-policy distillation extends the viable update horizon but eventually succumbs to the same failure mode. RL inserts capabilities efficiently but struggles to encode explicit factual knowledge. The practical ceiling forces labs to periodically retrain from a new base rather than indefinitely updating a single persistent model.