Value-driven Hindsight Modelling

Arthur Guez, Fabio Viola, Theophane Weber, Lars Buesing, Steven Kapturowski, Doina Precup, David Silver, Nicolas Heess

TL;DR: Reinforcement learning agents usually estimate the value of a state either by predicting the entire future (model-based) or by relying on a single delayed reward signal (model-free). This paper introduces "Hindsight Modelling" (HiMo), a method that finds a highly efficient middle ground: it learns to predict only the specific features of the future that are most useful for estimating the final return, significantly accelerating learning and boosting performance on complex benchmarks like Atari 57.

The Problem: Extremes in Value Estimation

In reinforcement learning (RL), accurately estimating the "value" (expected future return) of a state is critical. Currently, the field relies on two main paradigms, both of which have significant flaws:

  1. Model-free RL updates values based on a single scalar signal (the reward/return). This signal is often noisy, sparse, and ignores the rich, explanatory visual data present in the environment.
  2. Model-based RL tries to predict the entire future trajectory of the environment (e.g., generating future video frames). This is computationally expensive and forces the agent to waste capacity modeling irrelevant, chaotic details that have no bearing on the actual reward.

How can an agent leverage the rich information in future observations without the overwhelming burden of predicting every single pixel?

Value-driven Hindsight Modelling (HiMo)

The authors propose Value-driven Hindsight Modelling (HiMo). Instead of hand-crafting auxiliary tasks or predicting raw pixels, HiMo discovers what future features are worth predicting based purely on their usefulness for value estimation.

The architecture works in three main steps during training:

  1. The Hindsight Value Function ($v^+$): The agent is allowed to "cheat" by looking $k$ steps into the future trajectory. This privileged future information is passed through a low-dimensional bottleneck layer to create a compressed representation ($\phi$). Because it's a bottleneck, the network is forced to extract only the features most critical for predicting the final return.
  2. The Forward Model ($\hat{\phi}$): The agent trains a lightweight predictive model to guess this useful future representation ($\phi$) using only present information (no cheating).
  3. The Model-Augmented Value Function ($v^m$): The standard, non-cheating value function uses the predicted future features ($\hat{\phi}$) as an extra input to better estimate the return.

At test time, the agent doesn't have access to the future, so it relies entirely on its forward model ($\hat{\phi}$) and the augmented value function ($v^m$) to make decisions.

Key Insights: Bottlenecking the Future

If a pitcher throws a baseball, predicting the exact final score of the game from the moment of release is incredibly hard. However, predicting an intermediate event—like whether the batter swings—is much easier and highly correlated with the final outcome. HiMo automates the discovery of these intermediate "swing" events.

By forcing the hindsight value function to compress the future into a low-dimensional vector, the network naturally filters out irrelevant environmental noise and isolates reward-critical events. Mathematically, predicting a compressed, value-relevant feature of the future is often an easier learning problem than directly predicting the highly variable final return. This richer training signal leads to faster convergence and better state representations.

Results

HiMo significantly reduces value error compared to model-free and model-based baselines in simple policy evaluation settings.

The authors evaluated HiMo across different environments:

  • Portal Choice: A newly introduced 2D navigation environment requiring context-dependent associations and handling delayed rewards. HiMo reliably reached optimal behavior much faster than the vanilla IMPALA baseline. Analysis showed the model successfully learned to predict the identity of the goal room to solve a combinatorial context puzzle.
  • Atari 57: When integrated with the state-of-the-art R2D2 agent, HiMo provided a massive performance boost. After 200k gradient steps (about 1 day of training), the median human-normalized score jumped from 832.5% (baseline R2D2) to 965% (HiMo + R2D2). The mean score increased from 2818.5% to 2980%.
  • Excelling at Delayed Rewards: HiMo showed particularly strong improvements in games with delayed rewards, such as Bowling. The agent learned to predict the intermediate event of the ball hitting the pins, which served as a highly reliable stepping stone for predicting the final score.

Limitations and Future Work

  • Irreducible Stochasticity: If an environment is highly stochastic or chaotic, predicting the future features ($\phi$) might become just as difficult as predicting the empirical return directly, which would negate HiMo's advantages.
  • Hyperparameter Sensitivity: The approach requires choosing a horizon $k$ (how far into the future the hindsight model is allowed to look), which may need to be tuned depending on the specific dynamics of the task.

The authors suggest that HiMo's representation learning approach could be highly complementary to other hindsight techniques, such as Hindsight Credit Assignment (HCA), which focuses on understanding an action's specific contribution to a past outcome.

Summary updated

Abstract

Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn value predictors from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future, but receive a potentially weak scalar signal (an estimate of the return). We develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end, we determine which features of the future trajectory provide useful information to predict the associated return. This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.
BibTeX
@inproceedings{guez2020value,
  author = {Arthur Guez and Fabio Viola and Theophane Weber and Lars Buesing and Steven Kapturowski and Doina Precup and David Silver and Nicolas Heess},
  title = {{Value-driven Hindsight Modelling}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=p1R2M9t9YM},
}
Figure