In standard supervised learning, we minimize a loss function by following its gradient. If the loss is convex and the learning rate is small, we are guaranteed to reach the bottom. Reinforcement Learning (RL) attempts to borrow this machinery, but it often breaks in spectacular ways. When we combine Temporal-Difference (TD) learning with non-linear function approximators like neural networks, the system frequently oscillates, diverges, or "explodes."
This instability is not a mere implementation detail; it is a fundamental mathematical friction between how TD updates and how neural networks learn.
The Semi-Gradient Problem
The core reason TD learning fails where supervised learning succeeds is that TD is not actually a gradient descent algorithm. In supervised learning, the target is fixed. In TD learning, the target depends on the current parameters of the model—a process called bootstrapping.
As David Brandfonbrener and Joan Bruna explain in Geometric Insights into the Convergence of Nonlinear TD Learning, TD updates are semi-gradients. We take the gradient with respect to the current state's value but ignore how the parameters affect the "target" value of the next state. Because of this, there is no fixed "loss surface" we are descending. Instead, Yann Ollivier characterizes TD as a fixed-point iteration in Approximate Temporal Difference Learning is a Gradient Descent for Reversible Policies. In non-reversible systems, this iteration can spiral away from the truth rather than settling on it.
Why Non-Linearity Triggers Divergence
For decades, researchers have known that TD works reliably with linear function approximation. John N. Tsitsiklis provided the foundational proof for this in Analysis of Temporal-Difference Learning with Function Approximation. In the linear case, the geometry of the feature space is fixed, ensuring the update remains a contraction toward a solution.
However, neural networks change their own "features" as they learn. This creates a dangerous feedback loop. As Justin A. Boyan and Andrew W. Moore 0001 demonstrated in Generalization in Reinforcement Learning: Safely Approximating the Value Function, even if the final value function is simple, the intermediate functions during learning can be highly discontinuous.
When the network updates to reduce the error at one state, it might inadvertently change the values of neighboring states in a way that increases the overall Bellman error. In multi-agent settings, Ling Pan notes in Regularized Softmax Deep Multi-Agent Q-Learning that this can lead to structural instability where weights grow without bound as the network amplifies its own overestimation errors.
Stale Gradients and Non-Stationary Targets
The instability is further compounded by the "stale" nature of information in deep RL. Traditional credit assignment tools like eligibility traces assume that features are constant. Dhawal Gupta and colleagues argue in From Past to Future: Rethinking Eligibility Traces that when the underlying neural network changes, the "backward view" of credit assignment contradicts the current state of the network, leading to updates that can actually decrease the accuracy of the value function.
Similarly, Yunpeng Zhai points out in Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation that high-capacity models like ResNets are particularly sensitive to the non-stationary targets created by TD losses. The rapid shifting of the target value causes the large encoder's parameters to oscillate, preventing the agent from ever learning a stable representation of the environment.
Mechanisms for Stability
If the theory suggests divergence is likely, why does deep RL work at all? Recent research has begun to identify the "stabilizers" that keep these systems on the tracks:
- Target Networks: By freezing the parameters used to calculate the target for a period of time, we temporarily turn the TD update into a standard gradient descent problem. Mattie Fellows and Shimon Whiteson prove in Why Target Networks Stabilise Temporal Difference Methods that this works by improving the conditioning of the TD update Jacobian, forcing a system that would otherwise diverge to become contractive.
- Overparameterization: Surprisingly, very large networks might be easier to stabilize. Yufeng Zhang and Zhaoran Wang show in Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory that in the "mean-field" regime—where networks are sufficiently wide—TD and Q-learning can globally minimize Bellman error and successfully evolve features despite the lack of traditional convergence guarantees.
- Residual Gradients: Instead of the standard TD update, one can minimize the Bellman Residual directly. As Mance E. Harmon and Leemon C. Baird III showed in Advantage Updating Applied to a Differential Game, using a residual gradient approach provides stable convergence even in continuous-time tasks where standard Q-learning fails.