Gamma-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction

Michael Janner, Igor Mordatch, Sergey LevineNeurIPS 2020 · OpenReview

In model-based reinforcement learning (RL), agents typically rely on "single-step" models to predict the immediate next state. To plan for the future, these models are "rolled out" sequentially. However, this approach introduces a fundamental trade-off: short horizons are insufficient for meaningful long-term planning, while long-horizon rollouts suffer from compounding errors, where minor inaccuracies early on snowball into massive deviations.

Value functions (like Q-functions) bypass this by predicting the total discounted reward over an infinite horizon. Yet, they are inherently tied to a specific reward function; if the task changes, the value function must be relearned from scratch.

This paper introduces the $\gamma$-model, a predictive dynamics model that bridges this gap. It offers the long-horizon benefits of a value function while remaining independent of specific task rewards, much like a standard dynamics model.

The $\gamma$-Model and Generative TD Learning

Instead of predicting just the next state ($s_{t+1}$), the $\gamma$-model predicts a distribution over all future states, weighted by a geometric distribution based on the discount factor $\gamma$. This allows the model to capture the entire discounted future of an environment in a single forward pass.

Comparison between standard single-step models and the $\gamma$-model's probabilistic infinite horizon.

To train this model, the authors propose Generative Temporal Difference (TD) Learning. Similar to how a value function uses the Bellman equation to bootstrap its estimates, the $\gamma$-model uses a recursive objective. The target distribution for a given state is a mixture of the immediate next state and a sample drawn from the model's own prediction at that next state.

The authors successfully implement this using two popular generative frameworks:

  1. Generative Adversarial Networks (GANs): Trained by minimizing an $f$-divergence between the model and the bootstrapped target.
  2. Normalizing Flows: Trained by directly regressing on log-densities, leveraging the tractable density evaluation of flows.

Key Insights and Advantages

  • Continuous Successor Representation: The $\gamma$-model acts as a continuous, generative analogue to the Successor Representation (SR). While traditional SR is limited to discrete states or expected feature vectors, $\gamma$-models enable full state-space generative predictions.
  • Constant-Time Prediction: Standard models require $O(n)$ operations to predict $n$ steps ahead. In contrast, a $\gamma$-model provides a comprehensive snapshot of the discounted future in a single $O(1)$ forward pass.
  • Shifting the Error Trade-off: By predicting the infinite horizon directly, $\gamma$-models eliminate test-time compounding errors during rollouts. Instead, they shift this burden to training-time bootstrap errors, aligning the challenge more closely with model-free RL.
  • Reward Independence: Because it predicts future states rather than rewards, a trained $\gamma$-model can be reused for new tasks within the same environment. The new value can be found simply by taking the expectation of the new reward function over the model's predicted state distribution.
Reweighting predictions and required model rollout lengths for different discount factors.

Performance and Results

Evaluated on continuous control benchmarks (Acrobot, Pendulum, Mountain Car, Reacher), $\gamma$-models accurately capture the discounted occupancy of a policy, effectively "seeing" through cycles and around corners in a single pass.

When integrated into an actor-critic algorithm ($\gamma$-MVE), the method matched the sample efficiency of state-of-the-art model-based methods like MBPO. Furthermore, $\gamma$-MVE converged significantly faster than pure model-free Soft Actor-Critic (SAC)—sometimes twice as fast—while maintaining high asymptotic performance.

Limitations and Future Work

While promising, the approach has limitations:

  • Dimensionality: Scaling these generative TD updates to high-dimensional inputs, such as raw images, remains a significant challenge.
  • Bootstrapping Stability: Like all TD-based methods, $\gamma$-models can experience instability during training due to the moving target of the bootstrap. More robust training techniques for generative bootstrapping are needed.
  • Policy Conditioning: Because the model is conditioned on the current policy, significant policy changes require the model to be updated, which is a standard hurdle in off-policy model-based RL.

Summary updated

Abstract

We introduce the gamma-model, a predictive model of environment dynamics with an infinite, probabilistic horizon. Replacing standard single-step models with gamma-models leads to generalizations of the procedures that form the foundation of model-based control, including the model rollout and model-based value estimation. The gamma-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the gamma-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.
BibTeX
@inproceedings{janner2020gamma,
  author = {Michael Janner and Igor Mordatch and Sergey Levine},
  title = {{Gamma-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=TPv_JdUUEgE},
}
Figure