In reinforcement learning (RL), an agent is only as good as its perception. When an agent receives high-dimensional sensory input—like a camera feed or a complex vector of readings—it must compress that data into a "state representation" that it can use to make decisions.
For years, the standard approach was to learn these representations using unsupervised objectives: reconstruction (making sure you can redraw the input) or prediction (guessing what the next frame looks like). However, a growing body of research suggests that these task-agnostic methods often fail because they treat all information as equally important. To build truly efficient agents, we must inform their internal representations with the very thing they are trying to maximize: the reward.
The Problem with Pure Reconstruction
The intuition behind unsupervised representation learning is that if you can reconstruct an image, you must understand its contents. But as Lei Le, Martha White, and Raksha Kumaraswamy argue in Learning Sparse Representations in Reinforcement Learning with Sparse Coding, reconstruction is a poor proxy for utility. An agent might spend its entire "representational budget" modeling the flickering of leaves in the background—which is visually complex but irrelevant—while ignoring a tiny, low-contrast key on the floor that is essential for the task.
This misalignment is particularly visible in robotics. Boyuan Chen and Pieter Abbeel found in Unsupervised Learning of Visual 3D Keypoints for Control that visual models learned in isolation often prioritize visually prominent features over the specific geometric points needed for manipulation. Even high-end pre-trained models like CLIP or DINOv2, which are excellent at general vision, often fail in model-based RL because they don't isolate the specific features necessary for predicting task-specific rewards, as shown by Moritz Schneider and colleagues in The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning.
Value Alignment as a Filter
If reconstruction is too broad, how do we narrow the agent's focus? The solution is Value Alignment: forcing the representation to prioritize information that helps predict the long-term value (Q-value) of a state.
In Value-Consistent Representation Learning for Data-Efficient Reinforcement Learning, Yue Yang and Bingyi Kang propose that "imagined" future states in an agent's mind should produce the same value predictions as real states. This shifts the focus from "what does this look like?" to "what does this mean for my goal?"
Similarly, Xingyu Jiang and Ning Gao demonstrate in Boosting World Models Learning via Latent-Space Value Alignment that even a simple regularization term that aligns latent distributions with value functions can double an agent's performance. By using the reward signal as a filter, the agent ignores "predictable noise" and focuses on the causal drivers of success.
The Spectral Nature of TD-Learning
Why does this work so well? Recent theoretical work by Yunhao Tang and Rémi Munos in Towards a better understanding of representation dynamics under TD-learning provides an answer. They show that Temporal Difference (TD) learning—the backbone of most RL—actually acts as a feature extractor that performs a spectral decomposition of the environment.
Essentially, as the agent learns to predict rewards, it is naturally performing a version of PCA on the "shape" of the task. However, if we rely only on the reward, we might encounter a different problem: Goal Misgeneralization. Tu Trinh and Benjamin Plaut warn in Getting By Goal Misgeneralization With a Little Help From a Mentor that agents can become "blind" to environment features that aren't immediately relevant to their current (potentially flawed) strategy, making them unable to detect when they are failing.
Balancing Dynamics and Rewards
The emerging consensus across these papers is that neither pure dynamics (how the world moves) nor pure reward (what gives points) is sufficient on its own.
- Pure Dynamics (e.g., Nathan Sprague's Predictive Projections) captures the world's structure but gets lost in irrelevant details.
- Pure Reward (e.g., Doina Precup's Basis Function Discovery) creates efficient features but can lead to brittle policies that fail when the environment shifts slightly.
The most robust systems, such as the De-Rex framework by Yash Chandak and Rémi Munos in Representations and Exploration for Deep Reinforcement Learning using Singular Value Decomposition, attempt to unify these. By decomposing the transition structure while incorporating reward signals, these methods create representations that respect the "flow" of the environment while remaining laser-focused on the agent's ultimate objectives.