Reinforcement Learning (RL) typically falls into two distinct camps: model-based agents, which learn how the world works to plan ahead, and model-free agents, which map states directly to actions to maximize reward.
While model-free agents (like DQN or PPO) excel in complex, high-dimensional spaces, they are notoriously brittle. They often overfit to specific reward signals; if the reward function changes or the agent moves to a slightly different environment, performance collapses. Model-based agents generalize better because they understand the underlying "physics" of the environment, but they are often computationally expensive and difficult to train.
This paper introduces DRIML (Deep Reinforcement and InfoMax Learning), a method that bridges this gap. By adding an auxiliary objective that maximizes the mutual information between current state-action pairs and future states, DRIML equips model-free RL agents with "model-like" predictive capabilities without the overhead of full environment modeling.
How DRIML Works
The authors propose augmenting a standard distributional RL algorithm (C51) with a self-supervised auxiliary task based on Deep InfoMax (DIM).
- The Predictive Objective: Instead of solely learning to predict rewards, the agent is trained to maximize the Mutual Information (MI) between its internal representation of the current state-action pair $(s_t, a_t)$ and its representation of a future state $s_{t+k}$.
- Contrastive Learning: The model utilizes an "InfoNCE" loss. The agent must learn to distinguish a "positive" future state (one that actually happened in its trajectory) from "negative" states (sampled randomly from other trajectories in the replay buffer).
- Multi-Scale Representations: The architecture uses a Convolutional Neural Network (CNN) to extract both "local" features (small patches of the screen) and "global" features (the whole scene). This allows the agent to track specific objects while understanding general environmental trends.
- Action-Conditioned InfoMax: Unlike previous work that only looked at successive frames, DRIML explicitly conditions the prediction on the action taken. This is crucial for RL, as the future state inherently depends on the agent's choices.
Adaptive Timescale (DRIML-ada)
A key innovation in this paper is that the "look-ahead" distance ($k$) is not fixed. The authors introduce a mechanism that automatically adjusts $k$ based on how predictable the agent's actions are. If the agent is doing something highly predictable (like moving in a straight line), it looks further into the future.
Key Insights and Results
The core insight of DRIML is treating predictive representations as an inductive bias. Even if a reward function changes, the physics of the environment usually stays the same. By forcing the agent to learn representations that are predictive of future states, it learns the "rules of the game" independently of the rewards. The authors also show mathematically that the speed at which the agent learns these representations is linked to the "spectral gap" of the environment's transition matrix—essentially, how predictable or stochastic the system is.
The researchers evaluated DRIML across a spectrum of environments, including synthetic Markov Chains, a 2D Ising Model, a continual learning variant of Ms. PacMan, and the Procgen Benchmark suite.
- Procgen Performance: DRIML significantly outperformed the C51 baseline and other contrastive methods like CURL. On the Procgen suite (designed specifically to measure generalization), DRIML-ada achieved a normalized score of 1.9, nearly doubling the performance of the standard C51 baseline (1.0).
- Faster Adaptation: In the Ms. PacMan continual learning task—where the "lethal" ghost changes every 5,000 episodes—DRIML adapted to new threats much faster than C51. Because DRIML was already tracking all ghosts to predict their movement, it didn't have to "re-learn" what a ghost was when it suddenly became dangerous; it only had to update its policy.
- Robustness to Noise: In the physics-based Ising model experiments, the DIM objective allowed the agent to successfully ignore random noise and focus on the structured, predictable parts of the environment.
Limitations and Future Directions
While the authors note the computational cost is minimal, adding auxiliary heads and processing contrastive batches does increase the memory and compute footprint compared to pure model-free RL. Furthermore, the performance of contrastive learning often depends heavily on the choice of negative samples and data augmentation strategies.
Looking forward, the authors suggest this approach could be extended to non-visual domains, such as natural language processing or robotics, where temporal consistency and predictability are equally important. The adaptive timescale mechanism could also be further refined using more sophisticated policy-likelihood estimators.
Summary updated