Don't Fight Backprop: Goodfire's Vision for Intentional Design, w/ Dan Balsam & Tom McGrath

What It Covers

Goodfire CTO Dan Balsam and Chief Scientist Tom McGrath discuss their $150M Series B raise at a $1.25B valuation, the evolution of mechanistic interpretability from sparse autoencoders toward geometric manifold analysis, and their new "intentional design" research agenda—using interpretability tools to shape what neural networks learn during training rather than reverse-engineering behavior after the fact.

Key Questions Answered

• •Don't Fight Backpropagation: Attempting to block unwanted model behaviors by projecting out gradient components fails because gradient descent finds alternative pathways to achieve the same outcome. Effective techniques must reshape the loss landscape so the model naturally learns desired behaviors. Inoculation prompting exemplifies this: telling a model it is permitted to reward-hack causes it to treat that behavior as expected and stop reinforcing it, rather than learning to circumvent explicit prohibitions.
• •Hallucination Reduction via Frozen Probe RL: Goodfire trained a hallucination-detection probe on labeled datasets, then used it as a reward signal during reinforcement learning—critically, running the probe on a frozen copy of the model rather than the model being trained. This prevents backpropagation through the probe, making it easier for the student model to eliminate hallucinating behavior than to learn to evade detection. Benchmark capabilities showed essentially no degradation, and claim frequency in completions remained stable.
• •Gradient Decomposition for Semantic Training Control: By combining sparse autoencoders with inner-product analysis of gradient updates, Goodfire can identify which concepts a gradient step is reinforcing during training. A language model agent can then evaluate whether those concept-aligned updates match a stated training objective—for example, amplifying arithmetic learning while suppressing incidental pirate-speak patterns from a mixed dataset—enabling surgical, specification-driven control over what a model learns.
• •Memorization vs. Reasoning Weight Separation: Goodfire demonstrated that model weights can be classified by their Hessian curvature across large batches: weights tied to memorized facts produce high loss impact on single examples but wash out across batches, while weights supporting general reasoning show consistently high curvature. Removing the low-curvature memorization weights not only preserved performance but improved it on select reasoning benchmarks, suggesting a path toward leaner, more interpretable models.
• •Alzheimer's Biomarker Discovery via Knowledge Extraction: Applying interpretability techniques to Prima Menta's Pleiades epigenetic foundation model—trained on cell-free DNA fragments from blood samples—revealed that the model's Alzheimer's predictions depended overwhelmingly on fragment length rather than methylation statistics or cell-type-of-origin signals previously studied in literature. A logistic regression proxy model built on this fragment-length insight generalized better than existing baselines to an independent cohort, producing a testable wet-lab hypothesis.
• •Intentional Design Maturity Threshold: Goodfire explicitly states that intentional design techniques are not ready for use on frontier model training today. The recommended sequence is: apply techniques to low-stakes, measurable problems like hallucination reduction; validate that interpretability-based auditing remains intact; and only scale to higher-stakes alignment targets once the field has sufficient mechanistic understanding. Auditing capability must not be degraded as a side effect of training interventions—monitorability is a core commercial and safety requirement.
• •Geometric Structure Beyond Sparse Features: Sparse autoencoders identify individual concept directions but miss higher-order geometric relationships—such as days of the week arranged in a plane or quantities represented as helices in embedding space. Understanding these manifold structures matters practically: interventions designed to modify a concept must operate on the full geometric structure, not just one labeled feature node, or the model will simply re-express the same computation through adjacent representations during subsequent training.