Introduction to latent variable energy-based models: a path toward autonomous machine intelligence

Anna Dawid, Yann LeCun

Abstract

Despite the remarkable achievements of Deep Learning (DL) and Reinforcement Learning (RL) in specialized domains, current artificial intelligence remains significantly more data-hungry and brittle than biological intelligence. This paper outlines a conceptual roadmap for achieving "autonomous machine intelligence" by transitioning from traditional probabilistic frameworks toward Latent Variable Energy-Based Models (EBMs). At the core of this proposal is the Hierarchical Joint Embedding Predictive Architecture (H-JEPA), a modular blueprint designed to enable AI systems to learn world models, reason, and plan complex actions with human-like efficiency.

The Limitations of Current AI Paradigms

Modern AI systems, while powerful, lack the common sense and predictive capabilities inherent in humans. A human teenager can learn to operate a vehicle in roughly 20 hours of practice, whereas a Reinforcement Learning agent often requires thousands of years of simulated experience to master similar motor tasks. The authors identify three fundamental gaps preventing the development of truly autonomous agents:

  1. Lack of World Models: Current systems do not possess an internal understanding of physics or causality. They cannot predict the consequences of their actions without direct, often catastrophic, trial and error.
  2. The Curse of Probabilities: In high-dimensional, continuous spaces—such as video sequences—probabilistic models become mathematically intractable. Attempting to normalize over every possible future outcome (the partition function) is computationally prohibitive.
  3. The Multimodality Problem: Predicting the future is inherently uncertain; there is rarely a single "correct" next state. Standard generative models often fail here, producing "blurry" results because they attempt to average all possible future outcomes into a single prediction.

Energy-Based Models and Latent Variables

To address these challenges, the paper advocates for a shift toward Energy-Based Models (EBMs). Unlike probabilistic models that must sum to one, EBMs assign a scalar "energy" value to pairs of inputs $(x, y)$. In this framework, low energy denotes compatibility or high likelihood, while high energy denotes incompatibility. This approach bypasses the need for expensive normalization, making it far more flexible in high-dimensional environments.

To handle the inherent uncertainty of the world, the authors incorporate Latent Variables ($z$). These hidden variables capture information about the environment not present in the initial observation, allowing the model to represent multiple possible outcomes for a single input. By varying the latent variable, the model can explore different potential futures without committing to a single, averaged prediction.

The Joint Embedding Predictive Architecture (JEPA)

The centerpiece of this theoretical framework is the Joint Embedding Predictive Architecture (JEPA). JEPA represents a departure from traditional generative models. While a generative model attempts to predict every pixel of a future frame—including irrelevant noise like the movement of leaves on a tree—JEPA predicts the representation of the future in an abstract feature space.

By focusing on representation rather than reconstruction, the system can ignore task-irrelevant details and focus on the underlying semantics of the scene. To prevent "representation collapse"—a common failure mode where the model produces constant, uninformative outputs—the authors utilize Variance-Invariance-Covariance Regularization (VICReg). This method ensures that the learned representations are informative, diverse, and decorrelated.

Reasoning as Energy Minimization

The paper proposes that "System 2" thinking—deliberate, slow reasoning and planning—can be framed as an optimization problem. In this view, reasoning is the process of finding a sequence of actions that minimizes the total energy (or cost) predicted by the world model.

Furthermore, the authors argue that hierarchy is essential for complex planning. Human intelligence operates at multiple scales (e.g., the high-level goal of "traveling to Paris" versus the low-level motor control of "stepping into a taxi"). The Hierarchical JEPA (H-JEPA) stacks multiple JEPAs to handle these varying levels of abstraction and timescales, allowing for sophisticated, multi-stage planning.

Key Insights and the Future of AI

Several key philosophical shifts underpin this work:

  • "SSL is the Cake": Following Yann LeCun’s famous analogy, Self-Supervised Learning (SSL) provides the vast majority of information bits required for intelligence (the cake), while Supervised Learning and Reinforcement Learning are merely the icing and the cherry, respectively.
  • Abandoning Pixels: The shift from generative models to predictive embedding models is the most significant technical "aha moment," allowing models to focus on relevant world dynamics rather than visual noise.
  • Safety through Intrinsic Cost: The architecture includes a "Cost" module, divided into hard-wired intrinsic costs and a trainable "critic." This provides a framework for building AI with guardrails that mimic biological signals like pain or hunger, ensuring safer autonomous behavior.

Conclusion

While components of this architecture, such as VICReg and basic JEPA models, have already demonstrated success in specific tasks, the integration of these parts into a fully autonomous H-JEPA system remains a "grand challenge." Building a system that seamlessly integrates a configurator, world model, and actor into a cohesive unit is the primary objective for the next decade of AI research. This roadmap moves the field away from simple pattern recognition and toward machines that can truly perceive, reason, and plan.

Summary updated

Abstract

Abstract Current automated systems have crucial limitations that need to be addressed before artificial intelligence can reach human-like levels and bring new technological revolutions. Among others, our societies still lack level-5 self-driving cars, domestic robots, and virtual assistants that learn reliable world models, reason, and plan complex action sequences. In these notes, we summarize the main ideas behind the architecture of autonomous intelligence of the future proposed by Yann LeCun. In particular, we introduce energy-based and latent variable models and combine their advantages in the building block of LeCun’s proposal, that is, in the hierarchical joint-embedding predictive architecture.
BibTeX
@article{dawid2024introduction,
  author = {Anna Dawid and Yann LeCun},
  title = {{Introduction to latent variable energy-based models: a path toward autonomous machine intelligence}},
  journal = {Journal of Statistical Mechanics Theory and Experiment},
  year = {2024},
  doi = {10.1088/1742-5468/ad292b},
}
Figure