The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering

What It Covers

Goodfire researcher Curt Tigges presents findings from the paper "Neurons as an Interpretable Basis," challenging the field's assumption that individual MLP neurons are too polysemantic to study directly. Using Qwen3.6 35B-A3B (5 million neurons across 40 layers, 256 experts per layer), Tigges demonstrates that neurons are more interpretable than previously believed, that 90% can be ablated with minimal performance loss, and that LLM agents can systematically characterize each neuron's function.

Key Questions Answered

• •Neuron Sparsity Threshold: Ablating 88% of neurons in Qwen3.6 35B-A3B by retaining only the top-64 activations per expert causes minimal benchmark degradation on non-reasoning tasks. On ARC-AGI puzzles, performance actually improves by 5–15 points, likely because the model reaches answers faster without internal noise. Reasoning tasks do degrade, but 100 million tokens of fine-tuning—achievable in one day on a few GPUs—restores performance to near-parity or better across math, code, and medical QA benchmarks.
• •SwigLU Architecture Enables Interpretability: Modern SwigLU MLPs feed two independent parameter matrices (gate projection and up projection) into each neuron, whose outputs are multiplied together. This conjunction mechanism allows neurons to encode context-conditional logic—for example, activating strongly only when both "cooking context" and "garlic ingredient" signals are present. This architecture, now standard in most frontier models, produces more interpretable neurons than older ReLU or GELU activations because it enables cleaner switching between distinct functional roles.
• •Neuron Scientist Agent Workflow: To characterize all 5 million neurons, Goodfire built an LLM agent called the Neuron Scientist. It collects thousands of activation examples per neuron, clusters them by similarity, generates paragraph-length hypotheses about each neuron's function, then iteratively refines those hypotheses using tools including counterfactual sentence generation, activation steering, and targeted probing. A separate validator model then predicts token-by-token activation strength from the description alone, and those predictions are patched into the original model to measure damage recovery.
• •74% KL Divergence Recovery as Validation Metric: The primary validation method patches a separate model's activation predictions—derived solely from the neuron's paragraph description—back into the original Qwen model. This recovers 74% of the KL divergence loss caused by fully ablating the neuron, weighted by how much each neuron mattered to the model's predictions. This metric provides a concrete, objective measure of description quality that goes beyond subjective labeling, and outperforms standard auto-interpret baselines by applying iterative frontier-model reasoning.
• •Neurons Function as Verbs, Not Nouns: Rather than treating neurons as static feature detectors (entity recognizers), the more accurate framing is that neurons perform control-flow operations—directing downstream computation toward or away from specific behaviors. Reasoning-specific neurons, for example, signal the model to increase confidence, pause, recheck, or pivot strategies. This verb-framing has practical implications: when monitoring models for safety-relevant behaviors like reward hacking, researchers should look for neurons that trigger behavioral shifts, not just those that represent concepts.
• •Expert Specialization Is Real and Measurable: Within Qwen's mixture-of-experts architecture, individual experts concentrate on one to three topics rather than mixing hundreds of concepts. Identified specializations include a legal/constitutional jurisprudence expert, a geometry expert, a timekeeping/units-of-time expert, and multiple domain-specific medical experts. This specialization means the router is performing meaningful semantic routing, and it opens a practical path for safety applications: localizing hazardous knowledge (e.g., WMD-related content) within specific experts and ablating or withholding those experts in public model releases.
• •Facts Live in Neuron Connections, Not Individual Neurons: A common misconception is that 5 million neurons cannot store more than 5 million facts. Facts are encoded in the weighted connections between neurons, not in individual neurons themselves. A community of just a few dozen neurons can represent hundreds of artist-to-movement mappings (e.g., impressionist vs. post-impressionist painters) because the number of possible activation pathways grows exponentially with network size. This means sparsification preserves factual recall well, as confirmed by near-parity performance on MMLU, GPQA Diamond, and MedQA after light fine-tuning.