When training reinforcement learning (RL) agents in complex environments, we often encounter a paradox: the real world is overflowing with visual and physical information, yet only a tiny fraction of it matters for the task at hand. If an agent is trained to navigate a bustling dock, its sensors will capture the movement of clouds, the ripples on the water, and the shifting shadows of nearby cranes.
Traditional model-based RL approaches attempt to build a perfect "digital twin" of the environment by minimizing observation reconstruction error. However, as Simon Shaolei Du argues in Denoised MDPs: Learning World Models Better Than the World Itself, models that prioritize perfect reconstruction often perform poorly because they waste precious representational capacity on these uncontrollable or reward-irrelevant distractors. M. M. Hassan Mahmud similarly notes in Constructing States for Reinforcement Learning that modeling every observation sequence leads to severe computational inefficiency and overfitting to environmental noise.
The solution is Value-Directed State Abstraction. Instead of compressing the world based on what it looks like, we compress the world based on what it is worth. By using the reward signal or the value function as an information bottleneck, an agent can completely ignore complex environmental dynamics that have no bearing on its goals.
The Value Function as an Information Filter
Why does value-directed compression work? The core intuition relies on a simple principle: if two distinct environmental states yield the exact same expected future rewards for all possible actions, they are functionally identical for decision-making. They can be collapsed into a single abstract state.
This insight is not new. In their foundational work Value-Directed Compression of POMDPs, Pascal Poupart and Craig Boutilier demonstrated that focusing exclusively on information necessary to predict future rewards allows for linear compression techniques that drastically reduce state space dimensionality without sacrificing decision quality. Even in model-free architectures with recurrent memory, such as the RL-LSTM introduced by Bram Bakker in Reinforcement Learning with Long Short-Term Memory, the pressure to maximize rewards naturally forces the network's internal memory cells to prioritize task-critical features over transient environmental details.
Mathematically, when we train an encoder to predict values rather than pixels, the gradient of the loss function with respect to task-irrelevant features drops to zero. If a moving cloud does not alter the expected return, the value network ignores it, and the encoder learns to drop it entirely from the latent representation.
[High-Dimensional Observation] (Pixels, Noise, Distractors)
│
▼
┌───────────────┐
│ Encoder │
└───────┬───────┘
│
▼
[Latent State Space] ──► Optimized ONLY to predict Value/Reward
│
▼
Filtered, Task-Relevant State
Mechanisms of Value-Based Compression
Modern research implements this information bottleneck through several distinct architectural mechanisms:
1. Value-Predictive Latent Dynamics
Instead of predicting future high-dimensional observations, agents can plan entirely within an abstract latent space optimized to predict rewards and state values. Junhyuk Oh et al. formalized this in Value Prediction Network, demonstrating that abstract states prioritizing reward prediction allow agents to plan effectively while remaining robust to stochastic visual noise.
This concept was extended to fully stochastic environments by Ioannis Antonoglou in Planning in Stochastic Environments with a Learned Model, showing that world models only need to capture the specific elements of environmental randomness that impact future values. Similarly, Tianyu Li et al. integrated reward signals directly into spectral learning frameworks in Efficient Planning under Partial Observability with Unnormalized Q Functions and Spectral Learning, bypassing unsupervised model learning entirely to create compact, task-biased representations for rapid planning.
2. Advantage-Centric and Control-Oriented Filtering
Sometimes, even predicting absolute rewards introduces unnecessary detail. What matters for control is the relative utility of choices. Gi-Soo Kim et al. show in Robust Tests in Online Decision-Making that centering a reward estimator on the relative advantage between actions filters out environmental variables that do not change the optimal decision.
When representational capacity is strictly limited, Evgenii Nikishin et al. propose using implicit differentiation in Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiation to force the model to allocate its limited capacity exclusively to transitions that dictate performance, ignoring dynamics that are easily predictable but irrelevant to control.
3. Explicit Statistical and Logical Feature Selection
Rather than relying solely on soft neural network biases, some frameworks explicitly prune the state space. Angela Zhou demonstrates in Reward-Relevance-Filtered Linear Offline Reinforcement Learning that applying thresholded LASSO regression directly to the reward function isolates reward-relevant features from exogenous noise far more accurately than applying general sparsity constraints to the entire transition model.
For high-dimensional planning with uncontrollable processes, Rohan Chitnis et al. introduce mask-learning in Learning Compact Models for Planning with Exogenous Processes to identify the minimal subset of exogenous variables that affect goal achievement. In the domain of symbolic AI, Samuel Yang-Zhao et al. utilize a sharpness criterion in A Direct Approximation of AIXI Using Logical State Abstractions to prioritize logical predicates that predict rewards, making theoretically complex agents computationally viable. This concept scales even to massive state spaces; Rolf A. N. Starre et al. show in Abstraction for Bayesian Reinforcement Learning in Factored POMDPs that automatically dropping entire state factors with minimal reward impact allows agents to plan efficiently in environments with state spaces as large as $10^{25}$.
Achieving Out-of-Distribution Generalization
The ultimate benefit of value-directed state abstraction is robustness to domain shifts. If an agent's representation ignores background colors or decorative objects because they don't affect value predictions, the agent will naturally generalize when deployed in a new environment with different background styles.
Several recent frameworks exploit this property to achieve robust generalization:
- Value-Consistent Alignment: Yang Yue et al. develop the Value-Consistent Representation (VCR) framework in Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning, aligning latent states based on their strategic Q-value distributions rather than raw visual features.
- Auxiliary Return Prediction: In What Effects the Generalization in Visual Reinforcement Learning: Policy Consistency with Truncated Return Prediction, Shuo Wang et al. introduce an auxiliary task requiring the agent to predict truncated future returns, forcing the internal representation to discard transient visual styles.
- Information Bottlenecks: Rui Yang et al. combine the Information Bottleneck principle with recursive action-value decomposition in Learning Robust Representations with Long-Term Information for Generalization in Visual Reinforcement Learning, discarding any visual feature that fails to contribute to long-term expected returns.
- Multi-Modal Filtering: When dealing with multiple sensors, Yangru Huang et al. integrate a reward-predictive function in Seek Commonality but Preserve Differences: Dissected Dynamics Modeling for Multi-modal Visual RL to ensure that both shared and unique sensor pathways filter out environmental noise that does not impact performance.
Task-Induced and Skill-Centric Abstractions
Value-directed abstraction can also be guided by prior tasks or available skills, allowing agents to reason over long horizons.
In Task-Induced Representation Learning, Jun Yamada et al. introduce TARP, which leverages data from prior tasks to train value-predictive encoders, doubling learning efficiency by ignoring background clutter. For unsupervised skill discovery, Jelena Luketina et al. present DIVRS in Learning Skills Diverse in Value-Relevant Features, evaluating value functions across source tasks to isolate features that drive performance, preventing the agent from learning useless, controllable behaviors.
Perhaps the most radical departure from visual representations is proposed by Dhruv Shah et al. in Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning. The authors construct a state representation entirely out of the value functions of the agent's available skills. The agent views the world not through pixels, but through the lens of "what can I achieve right now?" Because changes in lighting or decorative objects do not alter the success probability of underlying skills, the high-level policy becomes inherently immune to visual distractors.
Interestingly, filtering can sometimes be too aggressive. Guy Davidson et al. note in Investigating Simple Object Representations in Model-Free Deep Reinforcement Learning that while value-centric filtering is powerful, selectively removing representations of secondary reward sources can actually help an agent maintain focus on primary long-term goals, highlighting the delicate balance required when designing reward-centric abstractions.
By shifting the representation goal from descriptive accuracy to strategic relevance, value-directed state abstraction provides a mathematically grounded mechanism for agents to ignore the noise of the world and focus entirely on what matters.
Key Foundational Work
Reinforcement Learning with Long Short-Term Memory (2001) — contributed 1 concept elements to this direction.
Value-Directed Compression of POMDPs (2002) — contributed 1 concept elements to this direction.