In the standard deep reinforcement learning (RL) paradigm, we typically train agents end-to-end. A single gradient signal, derived from a sparse reward, travels all the way from the policy back through the visual or state encoder. While elegant, this approach forces the agent to solve two difficult problems simultaneously: learning what the world looks like (representation) and learning how to act within it (policy).
Recent research suggests that this coupling is a primary source of instability. When the policy changes, the distribution of data changes; if the representation also changes in response to those noisy policy gradients, the agent faces a "moving target" problem. By decoupling representation learning from the RL objective, we can create more stable, sample-efficient, and generalizable agents.
The Moving Target Problem
The fundamental issue with joint optimization is that RL gradients are often high-variance and non-stationary. Vincent Liu and Martha White argue in The Utility of Sparse Representations for Control in Reinforcement Learning that isolating features before they are used for incremental updates prevents the instability caused by non-stationary representations.
When a representation is updated by a policy gradient, the "meaning" of a state changes. For the policy, this is like trying to learn to drive while the windshield is constantly warping and the road signs are changing colors. This is particularly destructive in complex environments. For instance, Greg Wayne and colleagues introduced MERLIN Unsupervised Predictive Memory in a Goal-Directed Agent, which uses a stop-gradient operation between the policy and the memory system. This ensures that the agent's world model is shaped by predictive accuracy rather than the trial-and-error noise of the policy.
Dense Signals for Sparse Rewards
RL rewards are often "sparse"—an agent might wander for thousands of steps before receiving a single positive signal. If the representation depends solely on this signal, the encoder learns nothing during those thousands of steps.
Decoupling allows us to use auxiliary tasks—dense, self-supervised signals that don't require rewards—to shape the representation. Adam Stooke and Pieter Abbeel demonstrated in Decoupling Representation Learning from Reinforcement Learning that agents using frozen, reward-agnostic features can match or outperform end-to-end RL. They used Augmented Temporal Contrast (ATC) to train encoders to associate observations across time, providing a stable foundation for the policy to build upon.
Similarly, Zeyu Zheng and Satinder Singh found that even predicting random features Learning State Representations from Random Deep Action-conditional Predictions can create more useful features than those derived from sparse rewards, provided the reward signal is blocked from updating the encoder.
Stability through Freezing
A recurring mechanism across this research is the use of frozen encoders. By pre-training a visual or state encoder and then "locking" its weights, we provide the RL agent with a fixed coordinate system.
- Visual Stability: In Reinforcement Learning with Neural Radiance Fields, Danny Driess and Pete Florence use NeRFs to pre-train geometry-aware representations. Freezing these encoders prevents the policy's updates from destabilizing the 3D understanding of the scene.
- Object-Centric Reasoning: Rodion Vakhitov and Aleksandr Panov show in Object-Centric World Models Meet Monte Carlo Tree Search that using frozen, pre-trained object discovery modules provides a stable "visual vocabulary" that allows agents to focus on learning dynamics and planning.
- Robustness to Noise: Artemij Amiranashvili and Wolfram Burgard developed DRAP Pre-training of Deep RL Agents for Improved Learning under Domain Randomization, which pre-trains an encoder to "see through" visual noise. By the time the RL agent starts learning, it receives a clean, canonical signal, shielding it from the variance of domain randomization.
Generalization and Out-of-Distribution Performance
Decoupling isn't just about training speed; it's about robustness. When an encoder is trained end-to-end for a specific task, it often overfits to features that are only relevant to that reward. If the environment changes slightly (e.g., a red block becomes blue), the agent fails.
Andrea Dittadi and Frederik Träuble conducted a massive study The Role of Pretrained Representations for the OOD Generalization of RL Agents, training over 10,000 policies. They found that training on top of frozen, pre-trained representations is the key to handling "surprises." Because the representation was learned via a task-agnostic objective (like reconstruction or temporal consistency), it retains information about the world that might not be immediately necessary for the current task but is vital for generalizing to new ones.
Scaling to State-Action Representations
The benefits of decoupling extend beyond visual pixels to the very heart of RL algorithms. Scott Fujimoto and Shixiang Gu introduced SALE For SALE: State-Action Representation Learning for Deep Reinforcement Learning, which learns state and action embeddings independently of the reward signal. By blocking gradients from the Q-function to these encoders, their TD7 algorithm prevents the latent space from collapsing or overfitting to a specific policy, leading to massive performance gains in both online and offline settings.
In summary, decoupling representation from control treats "perception" as a stable service provided to the "actor." By shielding the representation from the turbulence of reinforcement learning gradients, we allow agents to learn from dense environmental signals, resulting in systems that are more stable, more efficient, and far more robust to change.