What are the key takeaways from this Machine Learning Street Talk episode?

Key insights include: **Energy-Based Models vs Neural Networks:** Energy-based models differ from traditional feedforward networks by applying cost functions to internal states, not just inputs and outputs. This requires two minimizations: one for the energetic minimum of hidden nodes and one for prediction error. Variational autoencoders exemplify this approach, with encoders, decoders, and cost functions operating on internal representations like Gaussian distributions.; **Agency Identification Problem:** Determining whether a system exhibits true agency versus sophisticated policy execution requires examining internal computations, not just observing behavior. An agent performing Monte Carlo tree search and planning can appear identical to a complex function transformation from outside. The practical approach involves measuring internal state sophistication using metrics like transfer entropy to assign degrees of agency.; **Test-Time Training Limitations:** Current test-time training methods train networks in supervised mode, then activate additional weight adjustments during deployment. This approach seems unwise because the original network never learned with those latent variables active during training. Traditional energy-based models optimize latent variables throughout the entire training process, not just at deployment, creating more robust learning.

What did Jeff Beck discuss on Machine Learning Street Talk?

Dr. Jeff Beck explores energy-based models, variational autoencoders, and the nature of agency in AI systems. The conversation covers geometric deep learning, Bayesian inference, self-supervised learning architectures like JEPA, continual learning challenges, and the future of autonomous AI systems capable of scientific discovery and experimental design. Key topics include: **Energy-Based Models vs Neural Networks:** Energy-based models differ from traditional feedforward networks by applying cost functions to internal states, not just inputs and outputs. This requires two minimizations: one for the energetic minimum of hidden nodes and one for prediction error. Variational autoencoders exemplify this approach, with encoders, decoders, and cost functions operating on internal representations like Gaussian distributions.; **Agency Identification Problem:** Determining whether a system exhibits true agency versus sophisticated policy execution requires examining internal computations, not just observing behavior. An agent performing Monte Carlo tree search and planning can appear identical to a complex function transformation from outside. The practical approach involves measuring internal state sophistication using metrics like transfer entropy to assign degrees of agency..

How long is this episode of Machine Learning Street Talk?

This episode is 46 minutes long. SignalCast provides an AI-generated summary so you can get the key insights in about 3 minutes.

Machine Learning Street Talk

VAEs Are Energy-Based Models? [Dr. Jeff Beck]

January 25, 2026

46 min episode · 2 min read

Jeff Beck

Episode

46 min

Read time

2 min

Topics

Design & UX, Artificial Intelligence, Software Development

AI-Generated Summary

Published Jan 25, 2026

Key Takeaways

✓Energy-Based Models vs Neural Networks: Energy-based models differ from traditional feedforward networks by applying cost functions to internal states, not just inputs and outputs. This requires two minimizations: one for the energetic minimum of hidden nodes and one for prediction error. Variational autoencoders exemplify this approach, with encoders, decoders, and cost functions operating on internal representations like Gaussian distributions.
✓Agency Identification Problem: Determining whether a system exhibits true agency versus sophisticated policy execution requires examining internal computations, not just observing behavior. An agent performing Monte Carlo tree search and planning can appear identical to a complex function transformation from outside. The practical approach involves measuring internal state sophistication using metrics like transfer entropy to assign degrees of agency.
✓Test-Time Training Limitations: Current test-time training methods train networks in supervised mode, then activate additional weight adjustments during deployment. This approach seems unwise because the original network never learned with those latent variables active during training. Traditional energy-based models optimize latent variables throughout the entire training process, not just at deployment, creating more robust learning.
✓Self-Supervised Learning Trade-offs: Joint embedding prediction architectures compress inputs and outputs into latent spaces for learning, avoiding pixel-level prediction requirements. The challenge is preventing mode collapse where both embeddings become zero. Non-contrastive methods like BYOL and Barlow Twins use various regularization techniques to maintain representation richness while avoiding the expensive negative sampling required by traditional contrastive approaches.
✓Continual Learning Requirements: True artificial intelligence requires systems that instantiate new objects or models when encountering unexpected situations, not just learning from fixed training sets. This involves Bayesian nonparametric approaches with Dirichlet process priors that trigger learning when surprises occur. Object-centered physics discovery enables systems to create brand new object representations autonomously to explain novel situations, combining existing modules in new ways.

What It Covers

Dr. Jeff Beck explores energy-based models, variational autoencoders, and the nature of agency in AI systems. The conversation covers geometric deep learning, Bayesian inference, self-supervised learning architectures like JEPA, continual learning challenges, and the future of autonomous AI systems capable of scientific discovery and experimental design.

Key Questions Answered

•Energy-Based Models vs Neural Networks: Energy-based models differ from traditional feedforward networks by applying cost functions to internal states, not just inputs and outputs. This requires two minimizations: one for the energetic minimum of hidden nodes and one for prediction error. Variational autoencoders exemplify this approach, with encoders, decoders, and cost functions operating on internal representations like Gaussian distributions.
•Agency Identification Problem: Determining whether a system exhibits true agency versus sophisticated policy execution requires examining internal computations, not just observing behavior. An agent performing Monte Carlo tree search and planning can appear identical to a complex function transformation from outside. The practical approach involves measuring internal state sophistication using metrics like transfer entropy to assign degrees of agency.
•Test-Time Training Limitations: Current test-time training methods train networks in supervised mode, then activate additional weight adjustments during deployment. This approach seems unwise because the original network never learned with those latent variables active during training. Traditional energy-based models optimize latent variables throughout the entire training process, not just at deployment, creating more robust learning.
•Self-Supervised Learning Trade-offs: Joint embedding prediction architectures compress inputs and outputs into latent spaces for learning, avoiding pixel-level prediction requirements. The challenge is preventing mode collapse where both embeddings become zero. Non-contrastive methods like BYOL and Barlow Twins use various regularization techniques to maintain representation richness while avoiding the expensive negative sampling required by traditional contrastive approaches.
•Continual Learning Requirements: True artificial intelligence requires systems that instantiate new objects or models when encountering unexpected situations, not just learning from fixed training sets. This involves Bayesian nonparametric approaches with Dirichlet process priors that trigger learning when surprises occur. Object-centered physics discovery enables systems to create brand new object representations autonomously to explain novel situations, combining existing modules in new ways.

Notable Moment

Beck challenges the assumption that physical embodiment defines agency, arguing a high-fidelity computer simulation of himself would only become an agent if placed in his physical body. He maintains agents must be physical entities, not just computational models, even when the simulated version performs identical calculations and exhibits indistinguishable behavior from outside observation.

Know someone who'd find this useful?

Episode Transcript

Geometric deep learning is a big part of, like, is a big part of the stack, if for no other reason than when we talk about, like, modeling the physical world. That means, like, incorporating the symmetries that exist in the physical world. So it's, like, we're highly motivated to employ a lot of those methods and techniques. But is the world written in code, or do you mean exploiting the regularities in the code that seem to have some Exploiting the regularities. No. It's it's like, look, we it things are it is the world is translation invariant. The world is, like, rotation well, not really because there's gravity. But, like, in principle, you know, there is a principle axis, but it's certainly rotationally invariant in the x y plane. Yeah. And if you if you wanna have a good model of the world as it actually is, it should incorporate those features. And, of course, you can discover it, you know, in a brute force y way, but the mathematician in me really wants to build build the symmetries in. And, fortunately, we've got a lot of great tools that were developed over the last several years that can do that. What's your view on agency? If I'm being, you know, like an FEP purist, I have to sort of say, like, oh, there's no difference between, you know, an agent and an object in in a very real way, or at least there's nothing structurally distinct between what how we model an agent and how we model an object. It's really just a question of of degrees. Right? An agent is is a really sophisticated object. Right? It has internal states that represent things over very long time scales. You know, it has sophisticated policies that are context dependent, which is basically saying really long timescales again, and things like that. Yeah. You know, there's the kind of the philosophical highbrow notion of agency that we introduce notions of, intentionality and self causation and things like that. I mean, the the really no nonsense version of an agency is it it's just it's just a thing which acts and performs some kind of computation. And I guess you could almost model anything as an agent, you know. Yeah. Well, so if if if your definition of an agent is something that executes a policy, then anything is an agent. Right? Iraq is an agent. Right? Every everything has you know, it's an input a policy is an input output relationship. When many people talk about agents, they they're adding a few they're adding, a few additional elements that I think have a lot to do with how the policy is computed. Right? So for example, when we think of how the difference between, like, us and, like like, really, like, amoebas, we we often cite things like planning, counterfactual reasoning, goal oriented behavior. Right? We're specifying things that that, have that that are specific mean that that …

Get the full transcript (9,722 words) + summary by email — free

One-time email with the complete transcript and AI summary of this episode. No account needed.

One email, no spam. We’ll also show you what SignalCast does.

Browse all Machine Learning Street Talk transcripts →