Trajectory Planning in Learned Latent Spaces

18 concepts12 papers1 proposal

When an autonomous agent—whether a humanoid robot or a virtual character—attempts to navigate a complex environment, it faces a fundamental representation problem. If the agent plans its future actions directly using raw sensory inputs like camera images, it must simulate how every pixel changes over time. As pointed out by Andreea Deac et al. and Garrett Thomas et al., pixel-level generation is computationally expensive and prone to compounding artifacts that are entirely irrelevant to the actual task logic.

Even when planning in configuration space rather than pixel space, high degrees of freedom (DOF) create an exponential explosion of possibilities—the curse of dimensionality. Evaluating thousands of "what-if" scenarios or long-horizon manipulation sequences becomes impossible in real time if the planner must search through a massive, unconstrained state space (Vincent Francois-Lavet et al.; Baiming Chen et al.).

The Core Intuition: Compressing Dynamics

The key insight behind this research direction is to separate perception from logic. Instead of planning in the raw observation space, an encoder compresses the high-dimensional input into a compact, low-dimensional latent space.

A well-learned latent space acts as an informational bottleneck. It strips away task-irrelevant details (like background textures or lighting changes) and retains only the essential features governing the environment's underlying physics and transition dynamics. By operating within these highly compressed spaces, planning algorithms evaluate transitions over abstract concepts rather than raw data streams.

The scale of this compression can be astonishingly aggressive:

  • Brian Ichter et al. demonstrate that a 50-dimensional humanoid robot state can be mapped to a mere 4-dimensional latent representation, turning computationally exhaustive global searches into tasks solved in seconds.
  • For even more complex systems, Jung-Su Ha shows that a 197-degree-of-freedom humanoid can successfully navigate obstacles using just a 3D latent state and a 1D latent command.
  • Jung-Su Ha et al. establish that high-dimensional configurations can be compressed into minimal manifolds while fully preserving the topological information necessary for collision-free path planning.

How Planning Operates in Latent Spaces

Once the environment's dynamics are mapped to a low-dimensional manifold, various planning paradigms can be deployed efficiently without the computational overhead of full-state simulation.

Sampling-Based and Tree Search Methods

Classical motion planning relies on sampling states to build roadmaps or trees (such as the Rapidly-exploring Random Tree, or RRT). Brian Ichter et al. combine deep learning autoencoders with these classical algorithms, allowing robots to efficiently sample valid paths directly within the latent space. To guide searches in high dimensions, Binghong Chen et al. execute value iteration within a compressed latent space, providing computationally efficient global guidance without the overhead of full-state discretization.

Neural and Value-Based Planning

Instead of explicit geometric sampling, reinforcement learning agents can plan implicitly by executing value algorithms over abstract states. Andreea Deac et al. utilize Graph Neural Networks to run the Value Iteration algorithm entirely within a latent space, focusing exclusively on state-value estimations and transition dynamics while avoiding image reconstruction.

Trajectory Optimization and Collocation

For long-horizon tasks, planning can be framed as optimizing a sequence of states. Oleh Rybkin introduces Latent Collocation (LatCo), which optimizes sequences of latent states rather than action sequences. By planning in the latent state space, the agent can temporarily "violate" physics constraints during optimization to discover high-reward regions, bypassing the optimization failures common to traditional action-shooting methods.

Scaling to Long Horizons and Abstract Concepts

As tasks grow longer and more abstract, latent planning provides a natural framework for hierarchical control and skill abstraction.

For complex, multi-step manipulations of deformable objects, Xingyu Lin et al. demonstrate that planning over learned short-term skills in a compressed latent space allows robots to efficiently optimize intermediate goals from visual inputs. To achieve real-time control in long-horizon robotic manipulation, Dongxiu Liu et al. utilize multi-layer perceptrons to perform backward planning within a compressed latent space, avoiding the massive computational overhead of video generation.

This principle extends beyond physical robotics. In the domain of image-to-image translation, Jing Hu et al. show that a low-dimensional "concept plan" can act as an intermediary abstraction that bridges high-dimensional image states and actions, making complex multi-step generations highly tractable for reinforcement learning agents.

Ultimately, by shifting the planning burden from high-dimensional observation spaces to structured, low-dimensional latent spaces, these frameworks enable agents to reason about long-term consequences efficiently, robustly, and in real time.

Key Foundational Work

Combined Reinforcement Learning via Abstract Representations (2019) — contributed 3 concept elements to this direction.
Robot Motion Planning in Learned Latent Spaces (2019) — contributed 2 concept elements to this direction.
Adaptive Path-Integral Approach for Representation Learning and Planning (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_space_planning_73a5097d,
  title = {latent space planning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-planning/rdr_73a5097d7e574ed18b1b6417f6d150ec},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_space_planning_73a5097d_v1,
  title = {latent space planning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-space-planning/rdr_73a5097d7e574ed18b1b6417f6d150ec/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake