The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

What It Covers

The winning team from the ARC-AGI-3 benchmark competition explains how they built a coding-agent harness using frontier LLMs to solve novel video games without instructions. The episode covers action efficiency scoring, LLM game priors, brute-force limitations, requirements-based engineering with AI coding agents, and whether benchmark performance correlates with genuine intelligence.

Key Questions Answered

• •Action Efficiency Scoring: ARC-AGI-3 scores agents by squaring the ratio of human baseline actions to AI actions per level, making even slightly inefficient solutions collapse toward zero. A system solving 60-70% of training games can still score only 36% because it takes two to three times more actions than the human median. Optimizing for level completion alone is insufficient — action economy must be a primary design constraint.
• •LLM Game Priors as a Shortcut: Frontier LLMs encode high-level game concepts like mazes, enemies, and goals from pretraining, giving them a significant head start over pure reinforcement learning approaches. Encoding pixel colors as named labels (e.g., "b" for blue, "g" for gray) rather than raw numbers measurably improves performance because it aligns game representations with the model's pretraining distribution, reducing out-of-distribution friction.
• •Brute-Force Prevention by Design: ARC-AGI-3 hardened against the stochastic goose approach that won the preview competition by making the timer bar deplete on any action — not just valid ones — and expanding the action space to over 4,000 mouse-click positions on a 64×64 grid. With games requiring hundreds to thousands of actions, the branching factor makes exhaustive search computationally intractable within competition time limits.
• •Hypothesis Lock-In as the Core Failure Mode: Both coding agents and game-playing agents share the same failure pattern: once they commit to a wrong hypothesis early, they rarely escape it. Agents frequently misidentify goals — treating energy bar depletion or repeated region stepping as win conditions. Harness design should include explicit mechanisms forcing agents to discard and regenerate hypotheses after a fixed number of failed attempts.
• •Requirements-Based Engineering with Coding Agents: The team uses formally numbered requirements with explicit test criteria before handing tasks to coding agents, rather than single-prompt vibe coding. Agents are instructed to flag requirement conflicts during implementation rather than approximate solutions. Post-implementation, agents verify each requirement is satisfied with specific textual evidence — a workflow that reduces hallucinated compliance and maintains code correctness across a rapidly expanding codebase.
• •Test-Time Training Constraints in Long-Context RL: Applying reinforcement learning to ARC-AGI-3 requires training over sequences of 100,000 to 200,000 tokens per game, far exceeding ARC-AGI-2 norms. The team uses reward shaping across 25-plus procedurally generated games, combining level-transition rewards, ARC score signals, code execution success, and reasoning-step length penalties. Training on shorter sequences and generalizing to longer ones is an active research challenge with no clean solution yet.