Abstract
While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.
Introduction
In deep reinforcement learning (RL), agents must simultaneously learn a representation of their environment and a value function to guide behavior. A significant challenge arises in sparse-reward environments, such as Montezuma’s Revenge, where the absence of external reward signals can lead to "representation collapse." Without gradients from a reward-based value function, the internal features of the agent fail to develop, stalling learning. Auxiliary tasks—secondary objectives like predicting future states or random signals—are commonly employed to mitigate this, yet the underlying mechanisms of how they shape representation dynamics have remained largely opaque.
Theoretical Framework: Learning Dynamics
This research shifts the focus from the steady-state properties of learned representations to the dynamics of the learning process itself. By modeling learning as a system of Ordinary Differential Equations (ODEs), the authors compare the trajectories of Temporal Difference (TD) learning against Monte Carlo (MC) methods.
The analysis reveals that TD learning inherently incorporates the environment's transition structure. Unlike MC methods, which update toward a target return, TD updates are bootstrapped, meaning a prediction at one state is updated toward the prediction at the next. This coupling forces the representation to align with the transition operator of the Markov Decision Process (MDP), even in the absence of rewards.
Spectral Decomposition and Auxiliary Tasks
A key contribution of this work is the formal link between learned features and the spectral properties of the environment. The authors demonstrate that the features induced by TD learning naturally align with the eigen-basis functions (EBFs) of the transition operator.
When an agent is tasked with an ensemble of auxiliary tasks (the "many-task limit"), the resulting representation converges to a subspace spanned by the top eigenvectors of the transition matrix. This leads to the definition of Resolvent Singular Basis Functions (RSBFs), which are optimal for approximating unknown reward functions. These RSBFs provide a "mathematical skeleton" of the environment, ensuring that the agent maintains a rich, structured representation regardless of reward sparsity.
Experimental Results
The theoretical insights were validated across several domains:
- Tabular MDPs: RSBFs demonstrated superior generalization across policy iteration compared to standard random features or basic eigenvectors.
- Arcade Learning Environment (ALE): In high-dimensional Atari games, agents utilizing Random Cumulants (RC)—an auxiliary task where the agent predicts random signals—showed significant performance gains. Notably, in Montezuma’s Revenge, the RC-augmented agent outperformed standard Double DQN and other ensemble baselines by maintaining representation diversity in the face of sparse rewards.
Conclusion
By establishing a principled link between the spectral properties of an environment and the dynamics of TD learning, this work transforms auxiliary tasks from empirical heuristics into theoretically grounded tools. The findings suggest that the choice of algorithm and the initialization of auxiliary tasks (e.g., using orthogonal subspaces) are critical for robust representation learning in complex, sparse-reward tasks.
Summary updated