Reward Driven State Abstraction in Reinforcement Learning

This direction is merged into another direction: task-relevant feature selection. Its last version stays readable and citable here.

Version 1 of 125 papers30 conceptsfrozen 2026-10-04

In a complex environment, an agent is bombarded with sensory data. If you are training a robot to navigate a warehouse, its cameras capture the position of crates, the flickering of overhead fluorescent lights, and the movement of clouds through a distant window. To a standard generative model, the clouds and the crates are both "features" to be reconstructed. But to a reinforcement learning (RL) agent, the clouds are irrelevant noise, while the crates are critical obstacles.

The research direction explored across these papers shifts away from modeling the entire environment toward reward-driven state abstraction. The core intuition is that the reward signal is not just a target to maximize, but a powerful filter that identifies which parts of the world actually matter for decision-making.

The Failure of Perfect Reconstruction

Traditional world models often rely on reconstruction: if an agent can recreate the pixels it just saw, it must understand the state. However, as Simon S. Du and Tongzhou Wang argue in Denoised MDPs: Learning World Models Better Than the World Itself, models that prioritize perfect reconstruction often perform worse. They waste capacity on "exogenous noise"—environmental dynamics that the agent cannot control and that do not affect its goals.

Instead of building a "perfect" map, Marcus Hutter suggests in Feature Markov Decision Processes ($\Phi$MDP) that state discovery should be treated as a compression problem. The "best" representation is the one that is as simple as possible while remaining sufficient to predict rewards. This principle allows an agent to ignore the "Noisy TV" problem—a classic RL trap where an agent becomes obsessed with unpredictable but irrelevant environmental changes.

Reward as a Selective Filter

How does an agent decide what to keep? Several researchers propose using the reward signal or the value function as a supervisor for the representation itself.

  1. Direct Reward Prediction: M. M. Hassan Mahmud demonstrates in Constructing States for Reinforcement Learning that building models based on their ability to predict future rewards, rather than full observation sequences, prevents agents from being distracted by environmental noise.
  2. Value-Relevant Features: In Learning Skills Diverse in Value-Relevant Features, Jelena Luketina and colleagues show that value functions can act as a relevance filter. By identifying state changes that influence expected returns across different tasks, the agent can distinguish between "controllable" noise and "functionally important" features.
  3. Action-Value Supervision: Rui Yang introduces ROUSER in Learning Robust Representations with Long-Term Information for Generalization in Visual Reinforcement Learning, which uses action-value functions to discard visual distractions. If a feature doesn't help predict long-term rewards, it is filtered out of the latent space.

Capturing Long-Term Dependencies

A major challenge in this direction is that some features are "implicitly" relevant: they don't provide a reward now, but they are necessary for a reward later. Hirotaka Hachiya and Masashi Sugiyama address this in Feature Selection for Reinforcement Learning by measuring the Conditional Mutual Information between state variables and the total discounted return. This ensures that variables influencing long-term outcomes are preserved, even if they have no immediate impact on the reward signal.

Similarly, Biwei Huang and Chaochao Lu propose Action-Sufficient state Representations (ASRs) in Action-Sufficient State Representation Learning for Control with Structural Constraints. By modeling the causal links between latent states and cumulative rewards, they ensure the agent focuses only on the minimal set of features necessary for optimal decision-making.

Information Bottlenecks and Task-Specific Novelty

In environments where rewards are sparse, agents often rely on "curiosity" to explore. However, Young-Jin Kim and Gunhee Kim argue in Curiosity-Bottleneck: Exploration By Distilling Task-Specific Novelty that "pure" curiosity is dangerous. Without a filter, an agent might explore a "Noisy TV" forever. By applying the Information Bottleneck principle, they force the agent to focus its curiosity only on observations that are predictive of the task goal.

This theme of "bottlenecking" information is echoed by Abhishek Gupta in Learning latent state representation for speeding up exploration, where multi-headed reward prediction across various tasks creates a shared representation that ignores task-irrelevant noise.

Biological and Semantic Foundations

This isn't just a computational trick; it may be how biological brains work. Maneesh Sahani suggests in A Biologically Plausible Algorithm for Reinforcement-shaped Representational Learning that neuromodulators like dopamine retroactively "weight" sensory updates. This ensures the brain prioritizes features that are statistically subtle but high in survival utility.

In practical applications like gaming, this manifests as semantic discovery. Junchi Liang shows in Task-Relevant Object Discovery and Categorization for Playing First-person Shooter Games that by correlating visual object clusters with changes in the agent's value function, an agent can autonomously learn to "see" only the enemies and items that impact its performance, effectively ignoring the complex but irrelevant background geometry of the game world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_reward_driven_state_abstraction_in_reinf_2d2615a2,
  title = {Reward Driven State Abstraction in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-driven-state-abstraction-in-reinforcement-learning/rdr_2d2615a2ad1b4a3d9d36be2823bfa82b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_reward_driven_state_abstraction_in_reinf_2d2615a2_v1,
  title = {Reward Driven State Abstraction in Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-driven-state-abstraction-in-reinforcement-learning/rdr_2d2615a2ad1b4a3d9d36be2823bfa82b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}