Planning autonomous behavior directly from high-dimensional observations, such as camera images or point clouds, presents a fundamental challenge: the search space is too vast. If an agent attempts to plan a sequence of actions by predicting future raw pixels, it wastes immense computational power modeling irrelevant details—like shifting shadows or background noise—rather than focusing on task-relevant dynamics.
To bypass this bottleneck, modern model-based reinforcement learning and robotics frameworks compress high-dimensional observations into a compact, low-dimensional latent space. By training a transition model entirely within this hidden feature space, agents can simulate thousands of potential futures in a fraction of the time.
Crucially, because these learned world models are parameterized as neural networks, they are fully differentiable. This allows planning to be framed not just as a random search, but as a continuous trajectory optimization problem, leveraging gradients to efficiently steer an agent toward its goal.
The Core Architecture: Planning via World Models
To plan in a latent space, an agent requires three core components: an encoder ($\phi$) that maps high-dimensional observations $o_t$ to a compact latent state $s_t$, a latent dynamics model ($f$) that predicts the next latent state given a current state and an action $a_t$, and a decoder or value predictor to evaluate the quality of the predicted trajectory.
Observation (o_t) ---> [ Encoder ] ---> Latent State (s_t)
|
+------+------+
| |
Action (a_t) v
| [ Dynamics Model ]
| |
+------------ v
Latent State (s_{t+1})
In the foundational work of Hafner et al., this architecture (PlaNet) demonstrated that an agent could learn a world model from pixels and use it to evaluate action sequences entirely within the latent space. By unrolling the dynamics model forward in time ($s_{t+1} = f(s_t, a_t)$), the agent projects a trajectory into the future without ever reconstructing intermediate images.
The primary advantage of this approach is robustness to visual complexity. For instance, in Contrastive Variational Reinforcement Learning for Complex Observations, the authors utilize contrastive objectives rather than pixel reconstruction. By maximizing the mutual information between latent states and observations, the model discards complex, task-irrelevant background noise, ensuring that the latent space only encodes the causal variables necessary for planning.
From Sampling to Gradients: Why It Works
Once a latent dynamics model is trained, how does the agent find the optimal sequence of actions?
Early methods relied heavily on derivative-free, sampling-based optimization, such as the Cross-Entropy Method (CEM) or Path Integral Control. As seen in Learning Predictive Representations for Deformable Objects Using Contrastive Estimation, an agent can sample a distribution of action sequences, simulate their outcomes using the forward model, and select the sequence that minimizes the distance to a goal embedding. Similarly, Zhao et al. extend this sampling paradigm to multi-robot coordination, where agents share their predicted latent trajectories to adapt to one another's intentions.
However, random sampling scales poorly as the planning horizon lengthens or the action space becomes high-dimensional. This is where the differentiability of the world model becomes transformative. Because the transition function $s_{t+1} = f(s_t, a_t)$ is a neural network, we can compute the analytical gradient of a target cost function with respect to the actions themselves.
Action Sequence (A) ---> [ Differentiable Latent Dynamics ] ---> Predicted Goal Distance (J)
^ |
| v
+----------------------- [ Backpropagate Gradients ∂J/∂A ] -----------+
Instead of guessing random paths, the agent uses backpropagation to calculate exactly how to tweak each action in a sequence to minimize the distance between the predicted final state and the target goal state.
- Pan et al. leverage this for zero-shot visual navigation, optimizing action sequences via gradient descent to minimize the distance to a target feature representation.
- Yamada et al. formulate motion planning as an iterative optimization process where latent variables are updated via gradient descent to balance multiple objectives simultaneously: reaching a target pose, maintaining valid kinematics, and avoiding obstacles.
- Das et al. take this a step further by differentiating directly through a visual Model Predictive Control (MPC) loop to perform inverse reinforcement learning, recovering cost functions from visual demonstrations.
Shaping the Latent Landscape
For gradient-based trajectory optimization to succeed, the geometry of the learned latent space must be well-behaved. If the latent space is highly non-convex, full of discontinuities, or plagued by local minima, standard gradient descent optimizers will get stuck.
To address this, researchers design specific loss functions that shape the latent landscape for planning. Jonsson et al. introduce the Minimum Action Distance (MAD) framework, which forces the Euclidean distance between two latent embeddings to approximate the minimum number of actions required to travel between them. This aligns the geometric structure of the latent space directly with the reachability dynamics of the environment, making distance-to-goal an exceptionally smooth and reliable cost function for gradient descent.
In manipulation tasks, Hung et al. demonstrate that optimizing paths directly within the continuous manifold of a Variational Autoencoder (VAE) allows robots to discover valid configurations and avoid obstacles without needing explicit inverse kinematics or geometric collision checkers.
Overcoming the Optimization Bottlenecks
While gradient-based planning is highly efficient, it suffers from two classic failure modes: the train-test gap (or exploitation of model inaccuracies) and local minima over long horizons.
Smoothing the Landscape
When an optimizer uses gradients to find an optimal action sequence, it can easily discover "adversarial" actions where the world model incorrectly predicts a high reward. This is known as the train-test gap. To solve this, Parthasarathy et al. introduce landscape smoothing techniques. By regularizing the optimization surface, they prevent standard gradient-based optimizers from exploiting model errors, allowing them to find highly successful trajectories in a fraction of the time required by brute-force search algorithms.
Coarse-Grained and Hierarchical Planning
If an agent plans at the level of primitive, low-level motor commands over a long time horizon, the optimization landscape becomes incredibly complex. A common solution is to plan over an abstract space of behaviors or skills rather than raw actions.
Long Horizon: [ Latent Skill 1 ] -------------> [ Latent Skill 2 ]
| |
v v
Low-Level: [a_1, a_2, a_3, a_4] [a_5, a_6, a_7, a_8]
Co-Reyes et al. introduce this hierarchical structure via a trajectory autoencoder (SeCTAR), where a high-level planner optimizes points within a continuous latent space of entire trajectories, which a low-level policy then executes. By planning in "skill space" rather than action space, Xie et al. show that the optimization horizon is drastically shortened, smoothing out the loss landscape and allowing gradient-based methods to solve complex, long-horizon locomotion and exploration tasks.
Key Foundational Work
Learning Latent Dynamics for Planning from Pixels (2019) — contributed 2 concept elements to this direction.
Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation (2019) — contributed 2 concept elements to this direction.
Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings (2018) — contributed 1 concept elements to this direction.