Task Relevant Latent Dynamics for Visual Reinforcement Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 190 papers90 conceptsfrozen 2026-10-04

In visual reinforcement learning (RL), an agent must transform a high-dimensional grid of pixels into a sequence of actions. For years, the dominant strategy for learning these representations has been reconstruction: training the agent to compress an image into a latent code and then expand it back into the original pixels. The logic was simple: if you can reconstruct the image, you must have captured everything important.

However, researchers have increasingly identified a "pixel trap." In a complex environment, most pixels are irrelevant to the task. A flickering television in the background, the movement of clouds, or the specific texture of a floor are visually salient but functionally meaningless. When we force a model to reconstruct every pixel, we force it to waste its limited "mental capacity" on these distractions.

The emerging research direction focuses on learning task-relevant latent dynamics—building world models that ignore the "visual junk" and focus exclusively on the features that govern rewards and transitions.

The Failure of Pixel Perfection

Standard world models, such as the early Dreamer architectures, rely on high-fidelity reconstruction to learn latent states. As noted by Maxime Burchi and Radu Timofte in MuDreamer: Learning Predictive World Models without Reconstruction, this objective treats all pixels as equally important. If an agent is trying to pick up a red cube, a reconstruction loss might prioritize the complex, shifting shadows on the wall over the tiny, critical gap between the robot's gripper and the cube.

This leads to what Hongyu Zang and colleagues call "exogenous noise" in Towards Control-Centric Representations in Reinforcement Learning from Images. When the environment contains information that the agent cannot control and that does not affect the reward, reconstruction-based agents become brittle. If the background changes—a common occurrence in real-world deployment—the agent’s latent representation shifts, and its policy collapses.

Learning What Matters via Dynamics and Rewards

If we stop reconstructing pixels, how does the agent learn what the world looks like? The key is to use the MDP structure itself as the supervisor.

Carles Gelada and Marc G Bellemare formalized this in DeepMDP: Learning Continuous Latent Space Models for Representation Learning. Instead of a decoder, they propose minimizing two simple losses:

  1. Reward Prediction: Can the latent state predict the immediate reward?
  2. Next-Latent Prediction: Can the current latent state and action predict the next latent state?

By optimizing for these, the agent learns a "DeepMDP"—a compressed version of the environment that preserves only the causal structure necessary for the task. This approach is echoed in MuDreamer, which achieves superior robustness by predicting task-relevant quantities like values and actions instead of raw pixels.

Bisimulation and Behavioral Similarity

A more radical approach to filtering noise is Bisimulation. The intuition is that two states should be considered identical if they lead to the same rewards and the same future states, regardless of how they look.

In Learning Invariant Representations for Reinforcement Learning without Reconstruction, Amy Zhang and Sergey Levine introduced Deep Bisimulation for Control (DBC). DBC trains the encoder such that the distance between two points in latent space corresponds to their "behavioral similarity." If a robot is at the same coordinates in two different scenes, but one scene has a different wallpaper, DBC maps them to the same latent point. This creates a representation that is naturally invariant to visual distractions.

Information Bottlenecks and Contrastive Learning

Another way to force an agent to ignore noise is to restrict the "bandwidth" of the representation. Jiameng Fan and Wenchao Li use this in DRIBO, applying a Multi-View Information Bottleneck. By maximizing the mutual information between different views of the same state (or sequential frames) while compressing the representation, the model is forced to discard features that are not consistent or predictive—effectively filtering out transient background noise.

Similarly, Jeongsoo Ha and Kyungsoo Kim propose Dr. G (Dream to Generalize) in Dream to Generalize: Zero-Shot Model-Based Reinforcement Learning for Unseen Visual Distractions. They replace reconstruction with dual contrastive learning and inverse dynamics. Inverse dynamics—asking "what action was taken to get from state A to B?"—is particularly powerful because it forces the model to focus only on the parts of the image the agent can actually influence.

Disentangling the World

Finally, some researchers argue that we shouldn't ignore the "noise" entirely, but rather isolate it. Xiang Fu and Pulkit Agrawal introduced Task Informed Abstractions (TIA), which splits the world model into a Task Model and a Distractor Model.

By using adversarial training to ensure the Distractor Model contains no information about rewards, TIA allows the agent to "see" the whole world but only "plan" with the relevant parts. This prevents the model capacity from being overwhelmed by complex backgrounds while still allowing the system to account for the full visual field.

Summary: Why It Works

The shift away from reconstruction works because it aligns the representation learning objective with the agent's ultimate goal: maximizing return.

By prioritizing "control-centric" features—those that are predictable, controllable, and reward-linked—these methods produce latent spaces that are:

  • Sample Efficient: They don't waste steps learning to "draw" the environment.
  • Robust: they are invariant to changes in lighting, textures, or background motion.
  • Generalizable: they focus on the underlying physics and logic of the task, which remain constant even when the visual style changes.

As Jan Robine suggests in Simple, Good, Fast: Self-Supervised World Models Free of Baggage, "less is more." By removing the baggage of pixel reconstruction, we allow world models to finally focus on the dynamics that actually matter for decision-making.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_task_relevant_latent_dynamics_for_visual_62d3fde9,
  title = {Task Relevant Latent Dynamics for Visual Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-relevant-latent-dynamics-for-visual-reinforcement-learning/rdr_62d3fde9ba0e4933a9cd117536f73e55},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_task_relevant_latent_dynamics_for_visual_62d3fde9_v1,
  title = {Task Relevant Latent Dynamics for Visual Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/task-relevant-latent-dynamics-for-visual-reinforcement-learning/rdr_62d3fde9ba0e4933a9cd117536f73e55/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}