Learning State Abstractions under Exogenous Observation Noise

144 concepts131 papers1 proposal

When humans navigate the world, they possess a remarkable ability to filter out sensory overload. If you are driving a car, you easily ignore a flickering billboard, a passing flock of birds, or clouds moving across the sky. You focus strictly on what matters: the road, the steering wheel, and surrounding traffic.

In contrast, traditional deep reinforcement learning (RL) agents and world models suffer from sensory overload. When trained on high-dimensional inputs like raw pixels, standard models often use maximum likelihood estimation or pixel reconstruction objectives. This forces them to try to model and predict every single pixel in the environment. As a result, when an environment contains complex, temporally correlated distractors that the agent cannot control—known as exogenous noise—the agent wastes valuable representation capacity. As noted by Du et al., capturing these distractors in the latent state consumes computational resources and obscures the underlying physics necessary for effective planning and control.

To solve this, a major research direction focuses on learning invariant state abstractions that mathematically separate the task-relevant "signal" from the high-dimensional "noise."

The Exogenous Block MDP Framework

To formalize this problem, researchers frequently use the Exogenous Block MDP (EX-BMDP) framework. In an EX-BMDP, the high-dimensional observation $x$ is generated by a combination of two hidden, independent latent components:

  1. The Endogenous State ($s_{endo}$): The core state variables that are directly influenced by the agent's actions and are responsible for generating rewards (e.g., the position of a robot arm or a car).
  2. The Exogenous State ($s_{exo}$): Environmental factors that evolve according to their own complex dynamics, completely independent of the agent's actions and rewards (e.g., background traffic, changing weather, or camera jitter).

As Efroni et al. point out, these exogenous distractors can be highly structured and temporally correlated, making them look deceptively like meaningful state transitions to a naive agent. Without a mechanism to isolate $s_{endo}$, sub-policies and world models overfit to visual clutter, leading to severe training instability and poor generalization (Abdulhai et al.; Zeng et al.).

Biological evidence suggests that filtering this noise is essential for cognitive flexibility; Popov shows that both humans and animals rely on selective attention to reduce environmental complexity and isolate low-dimensional hypotheses during learning. In machine learning, researchers achieve this filtering through three primary mechanisms: inverse dynamics, empowerment, and value-gradient weighting.

Mechanism 1: Action-Conditioned Invariance (Inverse Dynamics)

How do you force a neural network to ignore what it cannot influence? The most prominent tool is the inverse dynamics model.

An inverse dynamics model takes two consecutive high-dimensional observations, $x_t$ and $x_{t+1}$, and attempts to predict the action $a_t$ that caused the transition.

   Observation (xt)   ───┐
                         ├─► [ Encoder ] ─► Latent Representation ─► [ Predict Action ] ─► Predicted ât
   Observation (xt+1) ───┘

Why it works intuitively

Imagine a video of a robot arm picking up a block while a television plays in the background. The changes on the television screen ($s_{exo, t} \to s_{exo, t+1}$) happen completely independently of whether the robot moves left or right. If an encoder tries to preserve the television pixels in its latent representation, those features provide exactly zero mutual information about the robot's actions.

By training an encoder solely to preserve information that helps predict the action $a_t$, the network is forced to discard the television screen as pure noise. It retains only the features that are causally linked to the agent's actions—the robot arm and the block.

By training representations through this lens, frameworks like those proposed by Tomar et al. and Li et al. successfully isolate agent-controllable features from raw visual streams. To handle even more deceptive, multi-step exogenous patterns, Efroni et al. utilize multi-step inverse dynamics to provably eliminate complex exogenous paths.

Mechanism 2: Empowerment and Causal Structure

A closely related perspective leverages empowerment—a mathematical measure of an agent's potential control over its environment, defined as the mutual information between sequences of actions and future states.

Instead of passively observing the environment, an agent can learn representations by maximizing its ability to influence the latent state space. As Bastankhah et al. demonstrate, maximizing empowerment naturally forces a state representation to filter out uncontrollable noise. They provide formal proof that interaction-driven learning can yield robust, control-centric world models without ever needing an explicit reward function.

By framing the agent as an active scientist, the authors of Cao et al. use empowerment to discover causal structures. This prevents the agent from overfitting to high-dimensional background clutter that has no functional impact on its ability to manipulate the environment.

Mechanism 3: Value-Gradient and Reward Filtering

What happens if a feature is controllable, but completely irrelevant to the task? For example, a robot might be able to change the color of a smart-light bulb in the room, but doing so has nothing to do with its task of opening a door.

To filter out task-irrelevant features (whether controllable or exogenous), researchers use value-aware or reward-driven abstractions.

Instead of forcing a world model to predict the entire future observation space, the model can be regularized or weighted using the gradients of a task value function. For instance, Voelcker et al. introduced a loss function (VaGraM) that uses value function gradients to inform the world model exactly which parts of the dynamics "matter" for the task.

$$\text{Loss} \propto \nabla_s V(s) \cdot \text{Dynamics Error}$$

If a complex, non-linear background distraction has zero impact on the expected cumulative reward, its value gradient $\nabla_s V(s)$ is zero. The dynamics error for that feature is ignored, allowing a capacity-constrained model to focus entirely on task-critical transitions.

Similarly, Liu et al. explicitly disentangle environment information into four distinct latent categories based on their relationship to actions and rewards. By isolating reward-relevant factors from both controllable and uncontrollable noise, their framework ensures that agents remain robust against severe visual distractions like camera jitter or dynamic backgrounds.

Summary

When operating in the messy real world, learning a "pixel-perfect" model of reality is a losing battle. By shifting away from raw reconstruction and instead leveraging inductive biases like inverse dynamics (what can I control?), empowerment (what can I influence?), and value gradients (what impacts my goal?), modern representation learning algorithms successfully separate the signal from the noise. They allow agents to compress high-dimensional, noisy observations into clean, low-dimensional latent spaces—effectively building world models that are cleaner and more reliable than the raw observations themselves.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensional_observation_noise_29b3f03f,
  title = {high-dimensional observation noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-observation-noise/rdr_29b3f03f834b4956b22edecdb775e852},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_dimensional_observation_noise_29b3f03f_v1,
  title = {high-dimensional observation noise},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensional-observation-noise/rdr_29b3f03f834b4956b22edecdb775e852/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake