Learning Controllable Representations for Reinforcement Learning

18 concepts14 papers1 proposal

When a reinforcement learning (RL) agent operates in the real world, it is bombarded with sensory information. A standard agent trained to reconstruct its entire visual input spends significant network capacity modeling irrelevant details: clouds drifting across the sky, leaves blowing in the wind, or a television flickering in the background. This susceptibility to irrelevant environmental entropy is often called the "Noisy TV" problem.

To build robust, sample-efficient agents, a growing body of research focuses on controllability-based feature selection. Instead of learning to represent every pixel, these methods construct a representational bottleneck that filters the world through the lens of agency, retaining only the features that the agent can actively influence or change.


The Inverse Dynamics Bottleneck

How do you force a neural network to ignore a flashing, unpredictable background distractor? One of the most elegant solutions is to task the network with predicting the agent's own actions.

In Curiosity-driven Exploration by Self-supervised Prediction, Pathak et al. introduced an intrinsic curiosity module grounded in an inverse dynamics model. The core intuition is simple: if a visual feature changes independently of the agent's actions, it contains zero information about what action the agent just took. By training a feature extractor solely to predict the action $a_t$ given state $s_t$ and next state $s_{t+1}$, the network naturally discards any environmental dynamics it cannot influence.

[ State s_t ] -----\
                    +---> [ Feature Extractor ] ---> [ Action Prediction â_t ]
[ State s_t+1 ] ---/

This action-prediction constraint serves as a powerful filter across various control settings:


Maximizing Empowerment and Mutual Information

While inverse dynamics models look backward to infer actions, another prominent approach looks forward, optimizing representations to maximize the agent's potential influence over its future environment—a concept known as empowerment. Mathematically, this is achieved by maximizing the mutual information $I(A; S_{next})$ between the agent's actions (or goals) and the resulting latent states.

If a background element moves randomly, its mutual information with the agent's choices is zero. Therefore, maximizing empowerment forces the latent representation to focus exclusively on features where the agent has high leverage.

In Information Prioritization through Empowerment in Visual Model-based RL, Bharadhwaj et al. demonstrate that constraining latent representations to correlate strictly with action sequences allows model-based agents to remain highly sample-efficient even when operating against noisy, real-world video backgrounds.

Bastankhah et al. provide a formal mathematical foundation for this in Learning to Perceive the World Through Control: Empowerment-Based Representation Learning. They prove that interaction-driven empowerment optimization creates an immense separation in variance—up to three orders of magnitude—between control-relevant features and uncontrollable distractions.

This principle translates naturally to goal-conditioned reinforcement learning. In Variational Empowerment as Representation Learning for Goal-Conditioned Reinforcement Learning, Choi et al. show that jointly optimizing a policy and a discriminator under an empowerment objective forces the emergence of goal spaces that automatically ignore irrelevant state dimensions. To make this robust against spurious correlations, Cao et al. introduce causal structure learning into the empowerment framework in Towards Empowerment Gain through Causal Structure Learning in Model-Based Reinforcement Learning, allowing agents to actively discover causal control structures in noisy environments.


Disentangling the Controllable World

Rather than completely discarding non-controllable information, several frameworks explicitly split the latent space into two distinct, decoupled components: an action-dependent (task-relevant) space and an action-independent (background) space.

                  /---> Action-Dependent Latent Space (Agent & Tools)
[ Raw Observation ]
                  \---> Action-Independent Latent Space (Static/Noisy Background)

In Denoised MDPs: Learning World Models Better Than the World Itself, Du et al. demonstrate that an agent can identify its own body and tools simply by analyzing which parts of an observation change in direct response to its actions. This controllability-based filtering allows the world model to isolate the agent's sphere of influence from background elements that evolve independently.

This explicit separation provides significant advantages across different paradigms:


Changing the Geometry of Latent Space

When representations are grounded entirely in control, the underlying metric geometry of the latent space changes. Instead of measuring visual similarity (e.g., pixel distance), distances in the latent space begin to reflect functional or actionable similarity.

In Learning Actionable Representations with Goal-Conditioned Policies, Ghosh et al. introduce Actionable Representations for Control (ARC). ARC learns a latent space where the distance between two states corresponds directly to the difference in actions required to reach them. Consequently, physically massive but control-irrelevant variations (like a massive changing background texture) are compressed into a single point, while tiny, highly actionable features (like the position of a small key) are expanded.

This control-centric geometry can even emerge naturally without explicit mathematical constraints through competitive dynamics. In Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning, Sukhbaatar et al. demonstrate that when two internal agents engage in asymmetric self-play to challenge one another, the learned goal encoder naturally prioritizes controllable environmental features. Because the agents can only interact with and challenge each other through features they can manipulate, the system inherently filters out static or uncontrollable background noise.

By filtering the world through the boundaries of their own agency, these diverse frameworks demonstrate that learning what can be controlled is often the most effective way to discover what matters.

Key Foundational Work

Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018) — contributed 3 concept elements to this direction.
Curiosity-driven Exploration by Self-supervised Prediction (2017) — contributed 1 concept elements to this direction.
Learning Actionable Representations with Goal-Conditioned Policies. (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_controllability_based_feature_selection_3bd33b39,
  title = {controllability-based feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllability-based-feature-selection/rdr_3bd33b39340e46ddbef06b11e7862a4f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_controllability_based_feature_selection_3bd33b39_v1,
  title = {controllability-based feature selection},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/controllability-based-feature-selection/rdr_3bd33b39340e46ddbef06b11e7862a4f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake