Abstract
In this work, we study the use of the Bellman equation as a surrogate objective for value prediction accuracy. While the Bellman equation is uniquely solved by the true value function over all state-action pairs, we find that the Bellman error (the difference between both sides of the equation) is a poor proxy for the accuracy of the value function. In particular, we show that (1) due to cancellations from both sides of the Bellman equation, the magnitude of the Bellman error is only weakly related to the distance to the true value function, even when considering all state-action pairs, and (2) in the finite data regime, the Bellman equation can be satisfied exactly by infinitely many suboptimal solutions. This means that the Bellman error can be minimized without improving the accuracy of the value function. We demonstrate these phenomena through a series of propositions, illustrative toy examples, and empirical analysis in standard benchmark domains.
Introduction
At the heart of modern Reinforcement Learning (RL) is the value function—an estimate of the total reward an agent expects to receive from a given state. Because the true value is rarely known, researchers have long relied on the Bellman equation as a surrogate. This equation posits that the value of a state should equal the immediate reward plus the discounted value of the subsequent state. The difference between these two sides is known as the Bellman error (BE).
For decades, the assumption has been that minimizing Bellman error is a reliable proxy for minimizing Value Error (VE)— the actual distance to the ground truth. However, this paper demonstrates that this assumption is fundamentally flawed. A low Bellman error does not necessarily imply an accurate value function; in fact, it can often hide systematic biases and lead to poor agent performance.
Methodology
The study focuses on policy evaluation—measuring a specific policy's value using a fixed dataset—across standard continuous control benchmarks in the MuJoCo simulator (HalfCheetah, Hopper, Walker2d, Ant, and Humanoid). The authors compare three primary approaches:
- Bellman Residual Minimization (BRM): Directly minimizes the squared Bellman error.
- Fitted Q-Evaluation (FQE): An iterative approach using the Bellman equation as an operator to update values.
- Monte Carlo (MC): A baseline that estimates values by averaging actual rollouts, minimizing value error directly.
The Bias Cancellation Trap
A primary reason for the failure of Bellman error as a metric is the "cancellation of bias." Because the Bellman error is calculated as a subtraction ($Q - \gamma Q'$), systematic errors can mask each other. For instance, if a value function consistently overestimates by a large margin, the subtraction on both sides of the equation can result in a near-zero Bellman error despite a massive actual Value Error.
As shown in the analysis above, BRM can achieve a mean Bellman error significantly lower than other methods (0.41 vs 2.76 for MC), yet its actual Value Error is nearly ten times higher than FQE and over eight times higher than MC. The ratio of VE to BE for BRM is a staggering 56.93, illustrating how the metric fails to capture the true inaccuracy of the model.
Empirical Results on Benchmark Tasks
The breakdown of the Bellman error as a proxy is most evident in complex environments like Ant and Humanoid. In these tasks, the authors observed an inverse relationship: algorithms that minimized Bellman error most effectively often produced the least accurate value functions.
In the learning curves above, BRM (red) quickly drives the Bellman Error to near zero across all environments. However, the corresponding Value Error curves show that BRM's actual accuracy is often the worst among the tested algorithms, particularly in the Ant and Humanoid environments where its Value Error skyrockets while its Bellman Error remains flat. Conversely, FQE (green) maintains a higher Bellman Error but achieves much lower Value Error, likely due to the neural network's ability to generalize across states rather than overfitting the Bellman metric.
Conclusion
This research highlights a critical pitfall in reinforcement learning: the Bellman error is a poor replacement for Value Error. Due to bias cancellation and the non-uniqueness of solutions in finite datasets, minimizing Bellman error can lead to highly inaccurate value functions. Practitioners should be cautious when using Bellman error as a primary metric for model selection or performance monitoring, especially in off-policy settings where the data may not fully represent the target policy.
Summary updated