Reinforcement learning (RL) agents often struggle when faced with the "curse of dimensionality." Whether it is a robot navigating a room using raw camera pixels or a drone processing thousands of sensor readings, the sheer volume of data can overwhelm the learning process. To solve this, researchers have converged on a powerful strategy: instead of learning to act directly from raw input, the agent first compresses that input into a compact, structured "latent space."
By using generative models like Variational Autoencoders (VAEs), agents can strip away irrelevant noise—like the specific texture of a wall or the flicker of a light—and focus on the underlying features that actually matter for decision-making.
The Bottleneck as a Filter
The core intuition behind this research direction is the use of an information bottleneck. In papers like Reinforcement Learning Embedded in Brains and Robots, Cornelius Weber and colleagues argue that biological systems do not process raw sensory data for every action. Instead, they perform unsupervised pre-processing to create structured representations.
In artificial systems, this is typically achieved through an autoencoder. An encoder network maps high-dimensional data (like a $64 \times 64$ image) into a small vector (a "latent state"). A decoder then tries to reconstruct the original image from that vector. For the decoder to succeed, the latent vector must contain all the essential information about the environment. Hongjia Li and Qi Zhu demonstrate that this dimensionality reduction allows agents to manage complex cyber-physical systems that would otherwise be computationally intractable Deep Reinforcement Learning: Framework, Applications, and Embedded Implementations.
Dreaming and Planning in Latent Space
Once an agent has a compact representation of the world, it can do more than just react—it can simulate. In the seminal World Models, David Ha and Jürgen Schmidhuber showed that an agent can learn a predictive model of how these latent vectors change over time. This allows the agent to "dream" or "hallucinate" future trajectories, training its policy entirely within its own internal simulation before ever attempting a task in the real world.
This "dreaming" capability is particularly useful for setting goals. In Visual Reinforcement Learning with Imagined Goals, Ashvin Nair and Vitchyr Pong introduced RIG, a framework where a robot uses a VAE to imagine what a "successful" state looks like. Because the goal is just a point in a low-dimensional latent space, the robot can easily calculate how close it is to that goal, enabling it to learn complex manipulation tasks from raw pixels in just a few hours.
Sample Efficiency and Real-World Stability
The primary benefit of learning in latent space is sample efficiency. Learning from raw pixels is slow because the agent must simultaneously learn "what an object is" and "how to move it." By separating representation learning from policy learning, the agent can master the "what" through unsupervised observation and then focus entirely on the "how."
This efficiency is critical for physical hardware. Alex Kendall and the team at Wayve used this approach to teach a car to drive in the real world in under 30 minutes Learning to Drive in a Day. Similarly, Philip Becker-Ehmck and Jan Peters applied these techniques to quadrotors, using hierarchical VAEs to infer hidden variables like velocity from noisy sensor sequences Learning to Fly via Deep Model-Based Reinforcement Learning.
Structured Latents for Complex Domains
The utility of latent representations extends beyond simple visual-motor tasks:
- Multi-Agent Coordination: Jiajun Chai and Dongbin Zhao use VAEs to compress the observations of neighboring agents, creating a "shared language" that allows hundreds of agents to cooperate without being overwhelmed by data NVIF: Neighboring Variational Information Flow for Large-Scale Cooperative Multi-Agent Scenarios.
- Continual Learning: To prevent "catastrophic forgetting," Hugo Caselles-Dupré and David Filliat developed S-TRIGGER, which uses the reconstruction error of a VAE to detect when an environment has changed and triggers a "generative replay" of old latent states to keep the agent's memory fresh S-TRIGGER: Continual State Representation Learning via Self-Triggered Generative Replay.
- Soft Robotics: For robots with infinite degrees of freedom, Andrew Spielberg and colleagues used VAEs to compress velocity fields into a tractable latent vector, enabling the simultaneous optimization of a robot's physical shape and its controller Learning-In-The-Loop Optimization.
By mapping the chaotic, high-dimensional reality of the world into a smooth, manageable latent manifold, these researchers have provided a blueprint for agents that can learn faster, generalize better, and operate safely in the real world.