Imagine an autonomous vehicle trying to navigate a busy highway. Its cameras capture a high-dimensional stream of pixels: the position of the road, the movement of surrounding cars, but also the shapes of drifting clouds, the rustling of leaves on roadside trees, and a flickering digital billboard.
If we train the vehicle’s internal representation using standard self-supervised techniques—such as variational autoencoders that reconstruct the image or forward models that predict the next pixel frame—the neural network will waste massive capacity modeling the clouds, the leaves, and the billboard. These complex, highly dynamic visual elements are completely irrelevant to driving. This dilemma is known as the "Noisy TV" problem, or the challenge of exogenous noise: complex, temporally correlated environmental distractors that an agent cannot control or influence.
To build effective representations for control, an agent's internal model must separate the controllable aspects of the world from the uncontrollable background noise. A powerful, mathematically rigorous way to achieve this is to flip the standard predictive paradigm on its head using multi-step inverse dynamics.
The Single-Step Filter
Instead of predicting the future state from a current state and an action (forward dynamics: $s_t, a_t \rightarrow s_{t+1}$), an inverse dynamics model looks at a state transition and predicts the action that caused it ($s_t, s_{t+1} \rightarrow a_t$).
Why does this simple change act as an information filter? Consider the mutual information between the agent's actions and the environment. If a feature in the environment changes completely independently of the agent's choices—like the static on a television screen or a cloud moving across the sky—that feature contains zero information about the action. To minimize its prediction error, an inverse dynamics model is forced to ignore these action-irrelevant features, stripping them out of the learned latent space.
As demonstrated by Deepak Pathak et al. in Curiosity-driven Exploration by Self-supervised Prediction, this mechanism forms the core of the Intrinsic Curiosity Module (ICM). By mapping observations to a latent space that only captures features affected by the agent's actions, the model prevents curiosity-driven agents from getting trapped by uncontrollable noise. Similarly, Evan Shelhamer et al. show in Loss is its own Reward: Self-Supervision for Reinforcement Learning that this self-supervised auxiliary task forces the encoder to prioritize features directly influenced by control, accelerating policy optimization even when external rewards are entirely absent.
This representational filtering is highly effective for structuring exploration. For instance, the authors of RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments combine forward and inverse models to ensure that exploration bonuses are driven purely by meaningful, agent-directed impacts on the environment. Mikael Henaff et al. leverage inverse dynamics embeddings in Exploration via Elliptical Episodic Bonuses to construct exploration bonuses that remain completely unaffected by background animations. As Minwoo Kim notes in Exploration in deep reinforcement learning: A survey, this filtering mechanism ensures that curiosity is directed toward controllable aspects of the environment rather than random environmental fluctuations.
Standard Forward Model: [State s_t] + [Action a_t] -------> Predicts [Next State s_{t+1}] (Models everything, including noise)
Inverse Dynamics Model: [State s_t] + [Next State s_{t+1}] -> Predicts [Action a_t] (Ignores features independent of a_t)
The Multi-Step Generalization
Single-step inverse dynamics works perfectly when environmental noise is completely random from frame to frame. However, it struggles when the noise is temporally correlated.
For example, a background pedestrian walking across a street follows a highly predictable, non-random path over time. Over a single time step, a single-step model might accidentally correlate a specific agent action with the pedestrian's passive movement simply by coincidence. If the agent happens to turn left every time a background distractor moves right, a single-step model might conclude that the distractor's movement is part of the agent's action footprint.
To resolve this ambiguity, the framework must be extended across a longer horizon. Instead of predicting a single immediate action, the model is trained to predict the sequence of actions (or the multi-step path) that connects two observations separated by multiple time steps ($s_t$ and $s_{t+k}$).
Multi-Step Inverse Dynamics:
[Observation s_t] + [Observation s_{t+k}] ---> Predicts [Action Sequence: a_t, a_{t+1}, ..., a_{t+k-1}]
Yonathan Efroni et al. formalized this mathematical intuition in Provably Filtering Exogenous Distractors using Multistep Inverse Dynamics by introducing the Exogenous Block MDP (EX-BMDP) framework. They proved that multi-step inverse dynamics can completely isolate controllable state variables from complex, temporally correlated distractors.
The underlying intuition is elegant: because exogenous distractors evolve completely independently of the agent, their distribution over time remains invariant regardless of the specific action path the agent chooses to execute. By forcing the model to solve a multi-class classification problem—predicting the full sequence of actions taken to reach an observation from a starting point—the encoder is mathematically compelled to discard the exogenous noise. The noise provides no predictive power over the specific sequence of choices made by the agent over a multi-step horizon.
Building on this, Manan Tomar introduced ACRO in Agent-Controller Representations: Principled Offline RL with Rich Exogenous Information, demonstrating that multi-step inverse models can effectively isolate controllable state components in offline reinforcement learning, allowing agents to ignore rich exogenous information that would otherwise derail standard baselines. Furthermore, Anurag Koul et al. utilize multi-step inverse dynamics in PcLast: Discovering Plannable Continuous Latent States to discover plannable continuous latent states, ensuring that states close together in the learned latent space are physically reachable via valid action sequences.
Deconfounding Intent and Isolating Dynamics
Beyond filtering out background noise, grounding representations in inverse dynamics provides deep structural benefits to the latent space, particularly when learning from demonstrations or planning future paths.
In multitask imitation learning, an agent often struggles because an expert's actions are driven by unobserved, hidden intentions. David Brandfonbrener et al. demonstrate in Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitation that pretraining with an inverse dynamics objective effectively "deconfounds" this hidden intent. By using the observed state transitions to isolate the causal action, the representation focuses strictly on the underlying physical state of the environment rather than getting confused by unobserved goals that might otherwise lead to inconsistent action predictions.
Rather than just filtering out noncontrollable dynamics, some architectures explicitly isolate them into separate processing streams. Minting Pan et al. introduce this concept in Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Models. By using an inverse dynamics constraint, they force one latent branch to capture only agent-controllable dynamics, which naturally pushes external environmental changes into a separate, action-free branch. This allows the agent to "peek" into the future of uncontrollable elements (like predicting the trajectory of an oncoming car) while maintaining a clean, unpolluted representation of its own capabilities.
Quantifying Control through Empowerment
The ability to accurately decode actions from state transitions is deeply tied to the information-theoretic concept of empowerment—a measure of how much causal control an agent has over its surroundings.
As shown by Shakir Mohamed et al. in Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning, this can be framed as a variational information maximization problem, where an action decoder acts as a variational lower bound on mutual information. When an agent can easily predict which actions caused a state change, it indicates a high degree of control or mutual information between the agent and its surroundings. This insight is leveraged by Xiaocong Chen et al. in Intrinsically motivated reinforcement learning based recommendation with counterfactual data augmentation to drive purposeful, agency-focused exploration in sparse recommendation environments.
Ultimately, by shifting the representation learning objective from pixel-perfect reconstruction to action-conditioned causal consistency, these methods provide a robust foundation for control. As Siyuan Li 0003 et al. demonstrate in Robust Visual Imitation Learning with Inverse Dynamics Representations, representations grounded in inverse dynamics ensure that learned features remain invariant to severe visual perturbations, allowing robots to successfully execute policies even when their deployment environment looks radically different from their training demonstrations.
Key Foundational Work
Loss is its own Reward: Self-Supervision for Reinforcement Learning (2017) — contributed 1 concept elements to this direction.
Curiosity-driven Exploration by Self-supervised Prediction (2017) — contributed 1 concept elements to this direction.
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015) — contributed 1 concept elements to this direction.