Task-Oriented Abstractions for Visual Reinforcement Learning

98 concepts82 papers1 proposal

When training reinforcement learning (RL) agents from high-dimensional visual inputs like raw pixels, a natural first instinct is to build a world model using an autoencoder or a generative pixel-prediction loss. This approach provides a dense, self-supervised learning signal. However, as noted by Evan Shelhamer et al. in Loss is its own Reward: Self-Supervision for Reinforcement Learning, generative objectives often force models to allocate massive computational resources to representing task-irrelevant environmental details.

This phenomenon creates a severe capacity bottleneck. Neural networks have finite representational capacity. When an objective forces a model to reconstruct every pixel—including background textures, shifting shadows, or a flickering television screen—it wastes capacity on features that do not influence decision-making. Simon Shaolei Du et al. formalize this in Denoised MDPs: Learning World Models Better Than the World Itself, showing that pixel-level reconstruction diverts the model's capacity toward modeling irrelevant environmental complexity. This is often illustrated by the "noisy TV" problem, where an agent becomes captivated by unpredictable visual noise that has zero impact on the underlying task dynamics, as described by Max Schwarzer et al. in Data-Efficient Reinforcement Learning with Self-Predictive Representations.

To solve this, a prominent research direction focuses on task-oriented abstractions: learning latent representations that completely bypass pixel reconstruction, focusing instead on the functional dynamics and causal relationships necessary for control.

Defining What Matters via Value Equivalence

If an agent shouldn't reconstruct pixels, what should its internal representation capture? One powerful answer is to retain only the information necessary to predict future rewards and values.

The theoretical foundation for this shift is detailed by Christopher Grimm et al. in The Value Equivalence Principle for Model-Based Reinforcement Learning. Instead of trying to create a perfect replica of the environment, the value equivalence principle dictates that two states are equivalent if they yield the same updates to the agent's value functions under a set of policies. By training the latent space purely to support downstream value estimation, the agent filters out task-irrelevant environmental noise.

[High-Dim Observation] ---> (Encoder) ---> [Latent State] ---> (Dynamics Model) ---> [Predicted Next Latent]
                                                 |                                          |
                                                 v                                          v
                                         [Reward/Value]                             [Reward/Value]

Practically, this principle enables highly efficient continuous control. For instance, Nicklas Hansen et al. introduce TD-MPC in Temporal Difference Learning for Model Predictive Control, which learns a latent dynamics model that completely ignores raw pixels and focuses exclusively on predicting rewards and values. By bypassing the computational overhead of pixel reconstruction, the model concentrates its parameters entirely on the causal factors required for planning.

Similarly, Yoshua Bengio argues in Combined Reinforcement Learning via Abstract Representations that feature extraction should be driven by the necessity of predicting future states and rewards rather than reconstructing raw inputs, protecting the agent's limited mental capacity from exogenous noise.

Actionable Representations and Behavioral Invariance

Another way to build task-oriented abstractions without relying strictly on reward signals is to look at actions. What can the agent actually control?

Dibya Ghosh et al. propose Actionable Representations for Control (ARC) in Learning Actionable Representations with Goal-Conditioned Policies. ARC learns a latent space where the distance between two states corresponds to the difference in actions required to transition between them. If a background distractor changes but the required actions to navigate the environment remain identical, the latent representation remains unchanged.

To achieve this without reconstruction, researchers often leverage inverse dynamics models (predicting the action taken between two consecutive states). Amy Zhang et al. operationalize this using bisimulation metrics in Learning Invariant Representations for Reinforcement Learning without Reconstruction. Their approach groups observations based on behavioral similarity, allowing an agent to completely ignore complex background distractions like moving clouds.

By training the representation to only care about features that change as a direct result of the agent's actions—a concept also explored by Homanga Bharadhwaj et al. in Information Prioritization through Empowerment in Visual Model-based RL—the model inherently discards uncontrollable environmental noise.

Joint-Embedding and Contrastive Predictive World Models

How do we train these world models mathematically if we don't have a decoder to compute a pixel-level reconstruction loss? The field has converged on two primary mechanisms: Joint-Embedding Predictive Architectures (JEPAs) and contrastive learning.

Instead of predicting future observations at the pixel level, a JEPA projects both the current and future observations into a latent space and predicts the future latent embedding directly. Basile Terver et al. evaluate this paradigm in What Drives Success in Physical Planning with Joint-Embedding Predictive World Models?, demonstrating that predicting in latent space prevents the world model from exhausting its capacity on high-dimensional noise.

To prevent the latent space from collapsing into a trivial constant vector, contrastive and self-predictive objectives are used. Xiao Ma introduces CVRL in Contrastive Variational Reinforcement Learning for Complex Observations, maximizing the mutual information between latent states and observations rather than recreating complex backgrounds.

Similarly, Tung D. Nguyen et al. develop Temporal Predictive Coding (TPC) in Temporal Predictive Coding For Model-Based Planning In Latent Space, which forces the model to focus only on elements of the environment that are predictable over time, discarding unpredictable background noise. Jiameng Fan further refines this in DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneck by using an information bottleneck to compress away distracting visual details while retaining sequential control features.

Mitigating the Bottleneck When Reconstruction is Required

In some scenarios, explicit state modeling or partial reconstruction is still desirable to guide exploration, multi-agent coordination, or spatial reasoning. Even in these cases, unconstrained pixel-level reconstruction is avoided in favor of targeted or abstract generation.

For example, Kyungmin Kim 0002 et al. propose Segmentation Dreamer, which uses segmentation masks to reconstruct only task-relevant objects (like a robot arm or a tool) while ignoring background motion. In the context of multi-agent systems, Yi Shen shows in Enhancing Cooperative Multi-Agent Reinforcement Learning with State Modelling and Adversarial Exploration that filtering out redundant environmental data prevents joint-state models from distracting the agent's policy.

By constraining what the model is allowed or forced to reconstruct, these methods preserve the dense learning signal of generative modeling without suffering from its capacity-wasting side effects. Discarding uncontrollable or reward-irrelevant features smooths the optimization landscape, making planning computationally lightweight and robust to real-world visual noise.

Key Foundational Work

Combined Reinforcement Learning via Abstract Representations (2018) — contributed 1 concept elements to this direction.
Learning Actionable Representations with Goal-Conditioned Policies. (2018) — contributed 1 concept elements to this direction.
Loss is its own Reward: Self-Supervision for Reinforcement Learning (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reconstruction_based_representation_bott_f86cd88a,
  title = {reconstruction-based representation bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconstruction-based-representation-bottleneck/rdr_f86cd88a0cbb4572876d26876f280834},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_reconstruction_based_representation_bott_f86cd88a_v1,
  title = {reconstruction-based representation bottleneck},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reconstruction-based-representation-bottleneck/rdr_f86cd88a0cbb4572876d26876f280834/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake