Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes

Andréa Agazzi, Jianfeng Lu

Abstract

We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We first consider this problem under a certain scaling of the approximating function, leading to a regime called lazy training. In this regime, the parameters of the model vary only slightly during the learning process, a feature that has recently been observed in the training of neural networks, where the scaling we study arises naturally, implicit in the initialization of their parameters. Both in the under- and over-parametrized frameworks, we prove exponential convergence to local, respectively global minimizers of the above algorithm in the lazy training regime. We then compare this scaling of the parameters to the mean-field regime, where the approximately linear behavior of the model is lost. Under this alternative scaling we prove that all fixed points of the dynamics in parameter space are global minimizers. We finally give examples of our convergence results in the case of models that diverge if trained with non-lazy TD learning, and in the case of neural networks.

Introduction

Temporal-Difference (TD) learning is a cornerstone of reinforcement learning (RL), used to estimate value functions that predict future rewards. While its convergence properties are well-understood for linear function approximators, the use of nonlinear models like neural networks has historically been fraught with theoretical instability. In the 1990s, seminal work by Tsitsiklis and Van Roy demonstrated that nonlinear TD learning could diverge to infinity. Despite these warnings, modern Deep RL (e.g., DQN) successfully employs massive neural networks. This research bridges the gap between classical divergence theory and modern empirical success by analyzing TD learning through the lens of "lazy training" and "mean-field" regimes.

Methodology

The authors investigate the dynamics of TD learning in the large-width limit of neural networks, focusing on two parameter scaling regimes at initialization:

  1. Lazy Training (NTK Regime): By scaling the network output by a factor $\alpha$, the parameters move minimally during training. This effectively "flattens" the nonlinear manifold of the neural network, causing it to behave like a linear model near its initialization.
  2. Mean-Field Regime: Parameters are allowed to evolve significantly. The collection of neurons is modeled as a continuous distribution, and its evolution is analyzed using Vlasov-type transport equations.

The study employs Lyapunov stability analysis to demonstrate that the error "energy" in these systems monotonically decreases, ensuring convergence.

Schematic of the effect of linear scaling ($\alpha$) on the approximating function manifold. As $\alpha$ increases, the manifold $\mathcal{F}_W$ flattens near the initialization point $V(w(0))$, stabilizing the nonlinear model.

Key Insights and Results

The research yields several critical findings regarding the stability of nonlinear TD learning:

  • Scaling as a Stabilizer: The "lazy" scaling ($\alpha \to \infty$) acts as a mathematical stabilizer. It restricts the network to a nearly linear region, bypassing the conditions that typically trigger nonlinear divergence.
  • Global Optimality: In the mean-field regime, the authors prove that any stable fixed point reached by the dynamics must be a global minimizer, precluding the existence of "bad" local optima under proper initialization.
  • Exponential Convergence: In the lazy regime, error decreases exponentially ($e^{-kt}$) toward a global minimum in over-parametrized settings.
  • Resolving Classical Divergence: The authors empirically validated their theory using the "Divergent Spiral"—a classic 3D counterexample where standard TD fails. Under lazy scaling ($\alpha=100$), the model achieved perfect convergence.

Discussion and Future Work

While this work provides a robust theoretical foundation for on-policy evaluation in Markov Reward Processes (MRPs), several challenges remain. The current analysis assumes a deterministic ODE limit, averaging out the stochastic noise inherent in real-world RL training. Furthermore, extending these proofs to off-policy Q-learning, where the agent's policy changes dynamically, represents a significant open problem. Future research may focus on non-asymptotic bounds to determine the minimum network width required for these stability regimes to hold in practice.

Summary updated

Abstract

We discuss the approximation of the value function for infinite-horizon discounted Markov Reward Processes (MRP) with nonlinear functions trained with the Temporal-Difference (TD) learning algorithm. We first consider this problem under a certain scaling of the approximating function, leading to a regime called lazy training. In this regime, the parameters of the model vary only slightly during the learning process, a feature that has recently been observed in the training of neural networks, where the scaling we study arises naturally, implicit in the initialization of their parameters. Both in the under- and over-parametrized frameworks, we prove exponential convergence to local, respectively global minimizers of the above algorithm in the lazy training regime. We then compare this scaling of the parameters to the mean-field regime, where the approximately linear behavior of the model is lost. Under this alternative scaling we prove that all fixed points of the dynamics in parameter space are global minimizers. We finally give examples of our convergence results in the case of models that diverge if trained with non-lazy TD learning, and in the case of neural networks.
BibTeX
@inproceedings{agazzi2021temporal,
  author = {Andréa Agazzi and Jianfeng Lu},
  title = {{Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes}},
  booktitle = {MSML},
  year = {2021},
  url = {https://openreview.net/forum?id=jm0qn5ALYLK},
}
Figure