Natural Value Approximators: Learning when to Trust Past Estimates.

Zhongwen Xu, Joseph Modayil, Hado van Hasselt, André Barreto, David Silver, Tom SchaulNeurIPS 2017 · OpenReview

Abstract

Neural networks have a smooth initial inductive bias, such that small changes in input do not lead to large changes in output. However, in reinforcement learning domains with sparse rewards, value functions have non-smooth structure with a characteristic asymmetric discontinuity whenever rewards arrive. We propose a mechanism that learns an interpolation between a direct value estimate and a projected value estimate computed from the encountered reward and the previous estimate. This reduces the need to learn about discontinuities, and thus improves the value function approximation. Furthermore, as the interpolation is learned and state-dependent, our method can deal with heterogeneous observability. We demonstrate that this one change leads to significant improvements on multiple Atari games, when applied to the state-of-the-art A3C algorithm.

Introduction: The Inductive Bias Mismatch

In deep reinforcement learning (RL), we rely on neural networks to approximate value functions—predictions of the total expected reward an agent will receive from its current state. However, a fundamental friction exists between the nature of deep learning and the structure of RL problems. Neural networks are designed with a "smooth" inductive bias; they assume that similar inputs should yield similar outputs.

In contrast, value functions are inherently discontinuous, especially in environments with sparse rewards. As an agent approaches a goal, its expected value rises, only to drop instantly the moment the reward is collected. Standard neural networks struggle to "snap" to these sharp transitions, often resulting in smeared or inaccurate estimates. This mismatch can lead to significant errors in value estimation, hindering the agent's ability to learn effectively.

Comparison of value function approximation on a toy task. NVA (red) captures the discontinuous "spikes" of the ground truth (dashed black) far better than the smooth baseline (blue).

Natural Value Approximators (NVA)

To bridge this gap, we introduce Natural Value Approximators (NVA). Instead of forcing a single network head to learn these discontinuities, NVA incorporates the temporal structure of rewards directly into the architecture. The system maintains three distinct components to produce a final value estimate ($G^\beta_t$):

  1. The Direct Estimate ($V_t$): A standard value prediction based on the current observation $S_t$.
  2. The Projected Estimate ($G^p_t$): A recursive estimate that projects the previous value forward. It is calculated by taking the previous estimate, subtracting the reward just received, and adjusting for the discount factor. Essentially, it uses the Bellman equation to predict what the current value should be based on past knowledge.
  3. The Blending Coefficient ($\beta_t$): A learned, state-dependent gating mechanism that decides the degree to which the agent should trust its current observation versus its projected past.

By blending these two estimates, the network can use the projected estimate to handle sharp discontinuities that the smooth direct estimator would otherwise miss.

Experimental Evaluation and Results

We evaluated NVA across a variety of environments, including toy regression tasks and the comprehensive Atari-57 benchmark using the Arcade Learning Environment (ALE). NVA was integrated into the Asynchronous Advantage Actor-Critic (A3C) algorithm to test its impact on state-of-the-art performance.

NVA shows widespread performance gains over the A3C baseline across 57 Atari games, with a median human-normalized score increase from 68.5% to 93.5%.

Adding NVA to A3C resulted in a substantial performance boost. On the "human starts" evaluation, the median human-normalized score across all 57 games jumped from 68.5% to 93.5%. Beyond raw scores, NVA consistently produced lower Root Mean Squared Error (RMSE) in value estimation—in some toy scenarios, the error was an order of magnitude lower than the baseline.

Key Insights

  • Value as Scalar Memory: NVA treats the previous value estimate as a highly relevant form of memory. While general-purpose memory like LSTMs tracks high-dimensional hidden states, NVA focuses specifically on the expected future return, which proved more effective for RL tasks in our tests.
  • Handling Partial Observability: In "foggy" environments where current observations are unreliable, the blending gate learns to rely almost entirely on the projected estimate ($\beta \approx 1$). This allows the agent to maintain an accurate value track even when it is temporarily "blind."
  • Architectural Priors: The success of NVA suggests that incorporating mathematical relationships (like the Bellman equation) directly into network architecture is more efficient than simply increasing model parameters.

Limitations and Future Work

NVA is currently an "online" method because of its recursive nature; extending it to work with Experience Replay (off-policy learning) remains an open challenge. Additionally, while this work focused on state-values ($V$), the framework could theoretically be extended to action-values ($Q$), which is a promising avenue for future research.

Summary updated

Abstract

Neural networks have a smooth initial inductive bias, such that small changes in input do not lead to large changes in output. However, in reinforcement learning domains with sparse rewards, value functions have non-smooth structure with a characteristic asymmetric discontinuity whenever rewards arrive. We propose a mechanism that learns an interpolation between a direct value estimate and a projected value estimate computed from the encountered reward and the previous estimate. This reduces the need to learn about discontinuities, and thus improves the value function approximation. Furthermore, as the interpolation is learned and state-dependent, our method can deal with heterogeneous observability. We demonstrate that this one change leads to significant improvements on multiple Atari games, when applied to the state-of-the-art A3C algorithm.
BibTeX
@inproceedings{xu2017natural,
  author = {Zhongwen Xu and Joseph Modayil and Hado van Hasselt and André Barreto and David Silver and Tom Schaul},
  title = {{Natural Value Approximators: Learning when to Trust Past Estimates.}},
  booktitle = {NeurIPS},
  year = {2017},
  url = {https://openreview.net/forum?id=LU1YdfhxV8V},
}
Figure