Introduction
Deep Reinforcement Learning (RL) agents are notorious for overfitting to their training environments. While researchers have developed various techniques to improve generalization, the root causes of this overfitting have remained elusive. This paper uncovers a fundamental tension in temporal difference (TD) learning: the very mechanism that helps agents fit complex, "jagged" value functions also forces them to "memorize" training states, severely hurting generalization.
The Conflict: Fitting vs. Generalization
The authors identify a core conflict between two primary goals in RL:
- Fitting the Value Function: Value functions in RL are often discontinuous or non-smooth, such as when an agent receives a sudden reward at a specific state.
- Generalization: Agents need to apply what they have learned to new, unseen observations.
This tension is governed by interference—the degree to which an update at one state changes the agent's predictions for other states. High interference leads to broad generalization, which is beneficial for new states. Conversely, low interference means the agent treats states independently, effectively acting like a lookup table. While this is excellent for fitting jagged functions, it is detrimental to generalization.
Methodology
To analyze these dynamics, the researchers combine theoretical frameworks with empirical benchmarks:
- Spectral Analysis: By using the eigendecomposition of the environment's transition operator, they define "smoothness." Smooth functions correspond to large eigenvalues, while non-smooth ones correspond to small or negative eigenvalues.
- Continuous-Time Dynamics: The learning process is modeled as a differential equation to compare how Monte Carlo (MC) updates differ from TD updates.
- Update Rank: To measure interference in deep networks, the authors introduce a metric called "update rank." By performing a singular value decomposition on a matrix of updates across different states, they can quantify whether a network is generalizing (low rank) or memorizing (high rank).
- Distillation: They evaluate post-training distillation, where a "student" network is trained to mimic the final outputs of a "teacher" network trained via standard RL.
Key Insights
- TD Learning is "Non-Smooth First": A critical discovery is that TD learning converges faster on the non-smooth components of a value function. Because non-smooth errors contribute disproportionately to the TD loss, the agent is incentivized to fit these jagged parts early in training.
- Implicit Regularization Towards Memorization: This "non-smooth first" bias acts as a form of implicit regularization that actively discourages generalization. As training progresses, agents naturally move toward a memorization regime to minimize TD error.
- The Story of Update Rank: In dense-reward Atari games, the update rank starts low (indicating generalization) but climbs significantly as the agent begins to fit the complex value function. Furthermore, in actor-critic setups, the critic (which relies purely on TD) exhibits a much higher update rank than the actor, confirming that the TD objective is the primary driver of memorization.
- Distillation as a "Reset Button": Distillation is effective because the targets are stationary. The student network does not experience the volatile, non-smooth early targets of TD learning, allowing it to find a smoother representation that still captures the teacher's performance.
Empirical Results
The study utilizes standard RL benchmarks, including MountainCar, the Atari 2600 Suite, and ProcGen, to validate these insights.
- Generalization Gains: Distilled agents consistently outperformed their online "teachers" on ProcGen test sets. In several games, distillation significantly closed the generalization gap between training and unseen levels.
- Robustness and Consistency: Distilled policies were found to be more robust to Gaussian noise on their inputs and demonstrated more consistent behavior when interpolating between different observations.
Limitations and Future Work
While post-training distillation is a powerful remedy, it requires an additional step after the main training loop. The authors suggest several avenues for future work:
- Advanced Distillation: Leveraging more sophisticated distillation methods from existing literature could further improve the student's raw performance.
- Optimizer Dynamics: The choice of optimizer (e.g., Adam vs. SGD) plays a fundamental role in these dynamics and warrants deeper investigation.
- Algorithmic Interventions: Instead of relying on post-hoc distillation, future research could explore modifying the TD update rule itself to adaptively balance interference and fitting during the primary training phase.
Summary updated