On The Effect of Auxiliary Tasks on Representation Dynamics

Clare Lyle, Mark Rowland, Georg Ostrovski, Will Dabney

Abstract

While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.

Introduction

In Deep Reinforcement Learning (RL), auxiliary tasks—secondary objectives such as predicting future states or random rewards—are frequently employed to enhance performance. Despite their empirical success, a formal understanding of how these tasks shape internal representations remains elusive. A critical failure mode in RL is representation collapse, particularly in sparse-reward environments. When rewards are rare, an agent may learn a zero value function by collapsing its internal features to zero, effectively halting the learning process. This paper investigates how auxiliary tasks provide the necessary "scaffolding" to maintain healthy representations by forcing the agent to incorporate the environment's transition structure.

Methodology: Learning Dynamics and Spectral Analysis

The authors analyze the continuous-time dynamics (ODEs) of Temporal Difference (TD) learning rather than just the final converged state. This framework allows for a comparison between TD and Monte Carlo (MC) methods.

Comparison of learning dynamics for 1-step Temporal Differences (left) and Monte Carlo (right). TD dynamics exhibit a structured flow that incorporates transition information, unlike the simple sink of MC.

A key contribution is the connection between learned features and the spectral decomposition of the environment's transition operator. The authors define Eigen-Basis Functions (EBFs) as the eigenvectors of the transition matrix and Resolvent Singular Basis Functions (RSBFs) as features induced by random cumulants. By examining the "many-task limit"—where an agent handles an infinite number of auxiliary tasks—the authors derive closed-form solutions for the resulting representation subspaces.

Key Insights: TD as a Feature Miner

The act of performing TD learning (bootstrapping) naturally "mines" the environment's transition structure. Even in the absence of external rewards, predicting random signals forces the representation to align with the smoothest eigenfunctions of the transition matrix.

Evolution of representations in a four-room gridworld. Over time, the learned features (left columns) align with the spectral components of the transition operator (right column), such as $\lambda_5$ and $\lambda_{105}$.

This alignment ensures that the representation spans the state space effectively. Specifically, using an ensemble of value predictors causes the representation to align with EBFs, which are known for their strong generalization properties. Random cumulants ensure that the representation never shrinks to zero, preventing collapse in sparse-reward scenarios.

Experimental Results

The theory was validated across several domains, including a tabular Chain MDP and the Arcade Learning Environment (Atari 2600). To specifically test sparse-reward scenarios, the authors created "censored" versions of Atari games (e.g., Sparse Pong) where rewards are significantly reduced.

In Montezuma’s Revenge—one of the hardest sparse-reward Atari games—the agent using Random Cumulants (DDQN+RC) significantly outperformed standard DDQN. In Sparse Pong, the RC-based agent was often the only one capable of learning a successful policy, while others suffered from representation collapse. Features learned via these auxiliary tasks were shown to generalize better across the "value-improvement path," remaining useful even as the agent's policy changes.

Conclusion

This work provides a theoretical foundation for the role of auxiliary tasks in RL. By framing TD learning as a dynamical system, the authors show that auxiliary tasks prevent representation collapse by forcing alignment with the environment's spectral structure. While currently focused on the linear regime, these insights offer a path toward more principled auxiliary task design in deep RL.

Summary updated

Abstract

While auxiliary tasks play a key role in shaping the representations learnt by reinforcement learning agents, much is still unknown about the mechanisms through which this is achieved. This work develops our understanding of the relationship between auxiliary tasks, environment structure, and representations by analysing the dynamics of temporal difference algorithms. Through this approach, we establish a connection between the spectral decomposition of the transition operator and the representations induced by a variety of auxiliary tasks. We then leverage insights from these theoretical results to inform the selection of auxiliary tasks for deep reinforcement learning agents in sparse-reward environments.
BibTeX
@inproceedings{lyle2021effect,
  author = {Clare Lyle and Mark Rowland and Georg Ostrovski and Will Dabney},
  title = {{On The Effect of Auxiliary Tasks on Representation Dynamics}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2021},
  url = {https://openreview.net/forum?id=dlorKVVm9Pm},
}

Loading related content...

Figure