Introduction
In reinforcement learning (RL), the value function is central to decision-making, estimating the total reward an agent expects to accumulate from a given state. Because the "true" value is rarely known during training, practitioners frequently rely on the Bellman equation as a consistency check. If the estimated value of the current state equals the immediate reward plus the discounted value of the next state, the Bellman error is zero, and the estimate is presumed accurate.
Consequently, "Bellman error" is often used as a surrogate objective for value prediction accuracy. However, this paper demonstrates that this is a dangerous assumption: minimizing Bellman error can actually lead to higher value error, particularly in off-policy settings. The metric is easily "cheated" by biased estimates and incomplete data.
Methodology
To investigate the relationship between Bellman error and true value error, the authors employ a mix of theoretical proofs and empirical experiments. They compare three primary approaches to policy evaluation using expressive deep neural networks:
- Bellman Residual Minimization (BRM): Directly minimizes the squared Bellman error.
- Fitted Q-Evaluation (FQE): An iterative method (standard in Deep RL) that treats the Bellman equation as a regression target.
- Monte Carlo (MC): Minimizes the distance to actual rollout returns, serving as the "ground truth" value error.
The study utilizes standard continuous-control benchmarks from the MuJoCo simulator (HalfCheetah, Hopper, Walker2d, Ant, and Humanoid). To test off-policy performance, datasets were generated using a target policy mixed with varying levels of random noise. This setup reveals how the relationship between Bellman error and value error degrades as data becomes less representative of the target policy.
Key Insights
The authors uncover several fundamental reasons why Bellman error fails as a reliable proxy:
- Magnitude Hides Bias: Bellman error measures the difference between two estimates. If both estimates are biased in the same direction (e.g., both overestimate the true value), the errors cancel out. Thus, a highly inaccurate value function can exhibit a deceptively low Bellman error.
- Missing Transitions Break the Equation: Real-world datasets are finite. The authors prove mathematically that if even a single transition is missing from the dataset, there are infinitely many suboptimal value functions that perfectly satisfy the Bellman equation (zero error) while being entirely wrong about the true values.
- FQE's Hidden Strength: FQE often exhibits higher Bellman error than BRM but achieves lower value error. This occurs because FQE imitates the Bellman operator and leverages the neural network's ability to generalize across missing data, rather than forcing a zero-error solution on a sparse, incomplete dataset.
Empirical Results
The empirical findings strongly support the theoretical claims. As shown in the learning curves below, the method that focuses solely on minimizing Bellman error (BRM) consistently yields the worst actual value accuracy.
In single-trajectory experiments, the ratio of Value Error to Bellman Error can be massive—for instance, in the HalfCheetah environment, BRM's value error was found to be nearly 57 times larger than its Bellman error.
When evaluating off-policy data, the disconnect becomes even more severe. As noise levels increase, BRM's Bellman error remains low, yet its actual value accuracy collapses. Conversely, FQE remains much more robust, maintaining lower value error despite higher Bellman error.
Limitations and Future Work
This study primarily focuses on policy evaluation (estimating values for a fixed policy) rather than the full control problem (learning the optimal policy). While evaluation is a core component of RL, dynamics may shift when the policy is actively changing. Additionally, the experiments focus on deterministic environments to isolate the "double sampling" problem. Future work should explore how stochasticity further complicates the reliability of Bellman error.
Ultimately, the main takeaway is a call to action for the RL community: we need better objectives for off-policy RL that do not rely on this flawed proxy.
Summary updated