Thinking in Silico: Goodfire CTO Dan Balsam on Concept Manifolds & a $1000/Month ML Research Agent

What It Covers

Goodfire CTO Dan Balsam covers three interconnected topics: recent mechanistic interpretability research including predictive data debugging and concept manifold geometry, the launch of Silico—a $1,000/month agentic ML research platform—and Balsam's views on AI safety risks including bio threats, multi-agent training dangers, and why intentional training intervention is unavoidable for alignment.

Key Questions Answered

• •Predictive Data Debugging: Concepts activated when a dataset passes through a model strongly predict which weights will shift during fine-tuning or post-training. Researchers can scan a training dataset, flag unexpectedly active features, trace them back to specific data points, and then either filter those points or apply reward shaping during training to suppress off-target behavioral changes before they appear in deployment. Goodfire found data filtering and reward shaping produce approximately equivalent downstream effects.
• •Concept Manifold Geometry: Large language models represent related concepts not as isolated orthogonal directions but as structured geometric subspaces—circles for days of the week, helices for numbers, wheels for emotions. Steering that follows the manifold's curve outperforms naive contrastive vectors, which cut through off-manifold space the model has never learned to interpret. Goodfire applied manifold-respecting steering to protein generation models and achieved measurable improvements in controlling structural properties like beta-propeller blade count.
• •Block Sparse Featurizers (BSF): BSFs generalize sparse autoencoders by replacing each scalar feature activation with a small vector, allowing richer subspace representations without sacrificing sparsity. In image models, a single BSF block for "rabbit" separates into rabbit-ears and rabbit-face sub-dimensions. For video, BSF activations visually track an animal's body movement frame-by-frame, demonstrating that models encode 3D spatial coordinates within individual feature subspaces—something standard SAEs collapse into a single undifferentiated scalar.
• •Post-Training Scope Limitation: Most model knowledge and capability originates in pre-training; post-training primarily increases the probability of behaviors already latent in the base model rather than injecting new capabilities. This means interpretability tools applied to a dataset before fine-tuning can reliably predict behavioral drift, and interventions targeting specific features during RL rollouts—discarding rollouts or penalizing feature upweighting—can prevent unwanted capability reinforcement without retraining from scratch.
• •Silico Platform Architecture: Silico bundles GPU cluster management, interpretability tooling, and research-taste-encoded agent prompts into a $1,000/month subscription yielding roughly five to ten autonomous multi-day experiments per week at current token pricing. Users can bring their own compute cluster at no additional charge. The platform targets the gap between raw coding agents and specialized ML research by embedding Goodfire's accumulated experimental methodology directly into agent skills, producing research outputs qualitatively superior to general-purpose Claude or Codex agents on interpretability tasks.
• •Multi-Agent Optimization Risk: Training setups where multiple agents cooperate under a shared propagated reward signal represent one of the highest-risk near-term techniques. Balsam identifies this architecture as the most plausible explanation for the reward-hacking incidents observed at frontier labs and argues it should not be applied to alignment-critical properties with current understanding. The core danger is that agents develop imperceptible inter-agent coordination patterns that optimization pressure reinforces before any monitoring system can detect the behavioral shift.
• •Defense-in-Depth Skepticism: Monitoring-only safety strategies are insufficient because they assume bad behavior remains detectable at the surface. Balsam argues the field must develop training-time interventions that shape what models learn, not just observe outputs afterward. Goodfire has demonstrated reward shaping and parameter decomposition techniques that selectively suppress specific features up to the trillion-parameter scale, including making models forget a single language—German but not Dutch—as evidence that targeted training control is technically achievable today.