When a reinforcement learning (RL) agent operates in the real world, it is bombarded with sensory information. A standard agent trained to reconstruct its entire visual input spends significant network capacity modeling irrelevant details: clouds drifting across the sky, leaves blowing in the wind, or a television flickering in the background. This susceptibility to irrelevant environmental entropy is often called the "Noisy TV" problem.
To build robust, sample-efficient agents, a growing body of research focuses on controllability-based feature selection. Instead of learning to represent every pixel, these methods construct a representational bottleneck that filters the world through the lens of agency, retaining only the features that the agent can actively influence or change.
The Inverse Dynamics Bottleneck
How do you force a neural network to ignore a flashing, unpredictable background distractor? One of the most elegant solutions is to task the network with predicting the agent's own actions.
In Curiosity-driven Exploration by Self-supervised Prediction, Pathak et al. introduced an intrinsic curiosity module grounded in an inverse dynamics model. The core intuition is simple: if a visual feature changes independently of the agent's actions, it contains zero information about what action the agent just took. By training a feature extractor solely to predict the action $a_t$ given state $s_t$ and next state $s_{t+1}$, the network naturally discards any environmental dynamics it cannot influence.
[ State s_t ] -----\
+---> [ Feature Extractor ] ---> [ Action Prediction â_t ]
[ State s_t+1 ] ---/
This action-prediction constraint serves as a powerful filter across various control settings:
- Planning from Pixels: Paster et al. utilize inverse dynamics in their framework, Planning from Pixels using Inverse Dynamics Models, to ensure that goal-conditioned world models represent only the features mediating the relationship between states and actions.
- Provable Exogenous Denoising: To provide theoretical guarantees, Lamb et al. developed the AC-State algorithm in Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models. They proved that multi-step inverse models combined with an information bottleneck can perfectly isolate "control-endogenous" states, completely discarding complex exogenous noise.
- Long-Term Exploration: In Unlocking the Power of Representations in Long-term Novelty-based Exploration, Saade et al. leverage action-state masking to prevent agents from becoming trapped by uncontrollable stochastic elements during long-horizon navigation tasks.
Maximizing Empowerment and Mutual Information
While inverse dynamics models look backward to infer actions, another prominent approach looks forward, optimizing representations to maximize the agent's potential influence over its future environment—a concept known as empowerment. Mathematically, this is achieved by maximizing the mutual information $I(A; S_{next})$ between the agent's actions (or goals) and the resulting latent states.
If a background element moves randomly, its mutual information with the agent's choices is zero. Therefore, maximizing empowerment forces the latent representation to focus exclusively on features where the agent has high leverage.
In Information Prioritization through Empowerment in Visual Model-based RL, Bharadhwaj et al. demonstrate that constraining latent representations to correlate strictly with action sequences allows model-based agents to remain highly sample-efficient even when operating against noisy, real-world video backgrounds.
Bastankhah et al. provide a formal mathematical foundation for this in Learning to Perceive the World Through Control: Empowerment-Based Representation Learning. They prove that interaction-driven empowerment optimization creates an immense separation in variance—up to three orders of magnitude—between control-relevant features and uncontrollable distractions.
This principle translates naturally to goal-conditioned reinforcement learning. In Variational Empowerment as Representation Learning for Goal-Conditioned Reinforcement Learning, Choi et al. show that jointly optimizing a policy and a discriminator under an empowerment objective forces the emergence of goal spaces that automatically ignore irrelevant state dimensions. To make this robust against spurious correlations, Cao et al. introduce causal structure learning into the empowerment framework in Towards Empowerment Gain through Causal Structure Learning in Model-Based Reinforcement Learning, allowing agents to actively discover causal control structures in noisy environments.
Disentangling the Controllable World
Rather than completely discarding non-controllable information, several frameworks explicitly split the latent space into two distinct, decoupled components: an action-dependent (task-relevant) space and an action-independent (background) space.
/---> Action-Dependent Latent Space (Agent & Tools)
[ Raw Observation ]
\---> Action-Independent Latent Space (Static/Noisy Background)
In Denoised MDPs: Learning World Models Better Than the World Itself, Du et al. demonstrate that an agent can identify its own body and tools simply by analyzing which parts of an observation change in direct response to its actions. This controllability-based filtering allows the world model to isolate the agent's sphere of influence from background elements that evolve independently.
This explicit separation provides significant advantages across different paradigms:
- Visual Imitation: In SeMAIL: Eliminating Distractors in Visual Imitation via Separated Models, Wan et al. decouple environment dynamics to let an agent "imagine" and imitate expert demonstrations without being misled by moving background distractors.
- Zero-Shot Generalization: In Focus On What Matters: Separated Models For Visual-Based RL Generalization, Huang et al. employ an empowerment-based loss term to ensure that passive environmental elements are never misclassified as task-relevant foregrounds, yielding robust generalization to unseen backgrounds.
- Offline RL Robustness: Unpredictable environments can cause offline agents to rely on environmental "luck." To combat this, Yang et al. propose the Dichotomy of Control: Separating What You Can Control from What You Cannot framework. By enforcing a mutual information constraint, they ensure that the latent representation of the future remains independent of stochastic environment transitions, forcing the policy to rely strictly on actions within its control.
Changing the Geometry of Latent Space
When representations are grounded entirely in control, the underlying metric geometry of the latent space changes. Instead of measuring visual similarity (e.g., pixel distance), distances in the latent space begin to reflect functional or actionable similarity.
In Learning Actionable Representations with Goal-Conditioned Policies, Ghosh et al. introduce Actionable Representations for Control (ARC). ARC learns a latent space where the distance between two states corresponds directly to the difference in actions required to reach them. Consequently, physically massive but control-irrelevant variations (like a massive changing background texture) are compressed into a single point, while tiny, highly actionable features (like the position of a small key) are expanded.
This control-centric geometry can even emerge naturally without explicit mathematical constraints through competitive dynamics. In Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning, Sukhbaatar et al. demonstrate that when two internal agents engage in asymmetric self-play to challenge one another, the learned goal encoder naturally prioritizes controllable environmental features. Because the agents can only interact with and challenge each other through features they can manipulate, the system inherently filters out static or uncontrollable background noise.
By filtering the world through the boundaries of their own agency, these diverse frameworks demonstrate that learning what can be controlled is often the most effective way to discover what matters.
Key Foundational Work
Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning (2018) — contributed 3 concept elements to this direction.
Curiosity-driven Exploration by Self-supervised Prediction (2017) — contributed 1 concept elements to this direction.
Learning Actionable Representations with Goal-Conditioned Policies. (2018) — contributed 1 concept elements to this direction.