Deep Reinforcement and InfoMax Learning

Bogdan Mazoure, Remi Tachet des Combes, Thang Doan, Philip Bachman, R. Devon Hjelm

Abstract

We begin with the hypothesis that a model-free agent whose representations are predictive of properties of future states (beyond expected rewards) will be more capable of solving and adapting to new RL problems. To test that hypothesis, we introduce an objective based on Deep InfoMax (DIM) which trains the agent to predict the future by maximizing the mutual information between its internal representation of successive timesteps. We test our approach in several synthetic settings, where it successfully learns representations that are predictive of the future. Finally, we augment C51, a strong RL baseline, with our temporal DIM objective and demonstrate improved performance on a continual learning task and on the recently introduced Procgen environment.

Introduction

Reinforcement learning (RL) typically bifurcates into model-based and model-free approaches. While model-free methods are effective in high-dimensional spaces, they often lack the robustness and generalization capabilities of model-based agents. This "brittleness" stems from representations that are overly specialized for immediate reward signals, ignoring the underlying dynamics of the environment. Deep Reinforcement and InfoMax Learning (DRIML) bridges this gap by introducing a self-supervised auxiliary objective that encourages model-free agents to learn representations predictive of future states, effectively granting them "model-like" foresight without the overhead of full state generation.

The DRIML Framework

The core of DRIML is the augmentation of a standard RL agent (specifically the distributional C51 agent) with a Deep InfoMax (DIM) auxiliary task.

Mutual Information Maximization

The agent is trained to maximize the Mutual Information (MI) between the representation of a current state-action pair $(s_t, a_t)$ and a future state representation $s_{t+k}$. This is implemented via the InfoNCE loss, a contrastive learning technique. The model learns to distinguish "positive" future states (actual successors in a trajectory) from "negative" samples drawn from other trajectories or distant time points.

Multi-Scale and Adaptive Features

DRIML utilizes both global features (full-screen context) and local features (spatial patches) to capture environmental dynamics at different granularities. Furthermore, the DRIML-ada variant introduces an adaptive lookahead parameter $k$. Instead of a fixed temporal distance, the agent dynamically adjusts $k$ based on the predictability of its actions, allowing it to look further ahead in stable environments.

Figure 1: (a) The DRIML architecture showing the encoder, value head, and auxiliary DIM heads. (b) The contrastive sampling strategy used to maximize mutual information across trajectories.

Experimental Evaluation

The authors evaluated DRIML across diverse settings:

  • Synthetic Tasks: Markov chains and 2D Ising models were used to verify the objective's ability to extract structure from noise.
  • Ms. PacMan (Continual Learning): A modified version where ghost lethality changes periodically, testing knowledge reuse.
  • Procgen Benchmark: A suite of 16 procedurally generated games designed to measure generalization to unseen levels.

Results and Discussion

DRIML demonstrated significant improvements in both generalization and adaptation.

Procgen Performance

On the Procgen benchmark, DRIML-ada achieved a normalized score of 1.9, nearly doubling the performance of the C51 baseline (1.0). It also outperformed other contrastive methods like CURL and CPC, suggesting that predicting future representations is a more effective auxiliary signal for RL than simple image-based contrastive tasks.

Adaptive Dynamics

The adaptive lookahead $k$ plays a crucial role in this performance. As shown in Figure 2, the agent adjusts its lookahead and action frequency as it learns the environment's dynamics. In games like Maze, the agent maintains a higher average $k$, reflecting the predictability of navigation tasks compared to more reactive games.

Figure 2: (a) Evolution of action counts and (b) average lookahead k over training timesteps for various Procgen environments.

Robustness and Adaptation

In Ms. PacMan, DRIML successfully ignored non-controllable "Ising noise" and adapted faster to changing ghost dynamics. By learning movement patterns through the predictive objective, the agent could reuse knowledge even when reward structures shifted. The authors also provide a theoretical link between the convergence of the DIM loss and the "spectral gap" of the environment's transition matrix, suggesting that more structured environments facilitate faster representation learning.

Conclusion and Future Work

DRIML provides a powerful method for enhancing model-free RL with predictive representations. While it introduces some computational overhead due to auxiliary heads, the gains in generalization on benchmarks like Procgen are substantial. Future research directions include extending the adaptive lookahead mechanism to continuous action spaces and refining the theoretical estimation of environment complexity in deep learning contexts.

Summary updated

Abstract

We begin with the hypothesis that a model-free agent whose representations are predictive of properties of future states (beyond expected rewards) will be more capable of solving and adapting to new RL problems. To test that hypothesis, we introduce an objective based on Deep InfoMax (DIM) which trains the agent to predict the future by maximizing the mutual information between its internal representation of successive timesteps. We test our approach in several synthetic settings, where it successfully learns representations that are predictive of the future. Finally, we augment C51, a strong RL baseline, with our temporal DIM objective and demonstrate improved performance on a continual learning task and on the recently introduced Procgen environment.
BibTeX
@inproceedings{mazoure2020deep,
  author = {Bogdan Mazoure and Remi Tachet des Combes and Thang Doan and Philip Bachman and R. Devon Hjelm},
  title = {{Deep Reinforcement and InfoMax Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=ZOGF1wMyPn},
}
Figure