In model-based reinforcement learning (RL), agents typically rely on "single-step" models to predict the immediate next state. To plan for the future, these models are "rolled out" sequentially. However, this approach introduces a fundamental trade-off: short horizons are insufficient for meaningful long-term planning, while long-horizon rollouts suffer from compounding errors, where minor inaccuracies early on snowball into massive deviations.
Value functions (like Q-functions) bypass this by predicting the total discounted reward over an infinite horizon. Yet, they are inherently tied to a specific reward function; if the task changes, the value function must be relearned from scratch.
This paper introduces the $\gamma$-model, a predictive dynamics model that bridges this gap. It offers the long-horizon benefits of a value function while remaining independent of specific task rewards, much like a standard dynamics model.
The $\gamma$-Model and Generative TD Learning
Instead of predicting just the next state ($s_{t+1}$), the $\gamma$-model predicts a distribution over all future states, weighted by a geometric distribution based on the discount factor $\gamma$. This allows the model to capture the entire discounted future of an environment in a single forward pass.
To train this model, the authors propose Generative Temporal Difference (TD) Learning. Similar to how a value function uses the Bellman equation to bootstrap its estimates, the $\gamma$-model uses a recursive objective. The target distribution for a given state is a mixture of the immediate next state and a sample drawn from the model's own prediction at that next state.
The authors successfully implement this using two popular generative frameworks:
- Generative Adversarial Networks (GANs): Trained by minimizing an $f$-divergence between the model and the bootstrapped target.
- Normalizing Flows: Trained by directly regressing on log-densities, leveraging the tractable density evaluation of flows.
Key Insights and Advantages
- Continuous Successor Representation: The $\gamma$-model acts as a continuous, generative analogue to the Successor Representation (SR). While traditional SR is limited to discrete states or expected feature vectors, $\gamma$-models enable full state-space generative predictions.
- Constant-Time Prediction: Standard models require $O(n)$ operations to predict $n$ steps ahead. In contrast, a $\gamma$-model provides a comprehensive snapshot of the discounted future in a single $O(1)$ forward pass.
- Shifting the Error Trade-off: By predicting the infinite horizon directly, $\gamma$-models eliminate test-time compounding errors during rollouts. Instead, they shift this burden to training-time bootstrap errors, aligning the challenge more closely with model-free RL.
- Reward Independence: Because it predicts future states rather than rewards, a trained $\gamma$-model can be reused for new tasks within the same environment. The new value can be found simply by taking the expectation of the new reward function over the model's predicted state distribution.
Performance and Results
Evaluated on continuous control benchmarks (Acrobot, Pendulum, Mountain Car, Reacher), $\gamma$-models accurately capture the discounted occupancy of a policy, effectively "seeing" through cycles and around corners in a single pass.
When integrated into an actor-critic algorithm ($\gamma$-MVE), the method matched the sample efficiency of state-of-the-art model-based methods like MBPO. Furthermore, $\gamma$-MVE converged significantly faster than pure model-free Soft Actor-Critic (SAC)—sometimes twice as fast—while maintaining high asymptotic performance.
Limitations and Future Work
While promising, the approach has limitations:
- Dimensionality: Scaling these generative TD updates to high-dimensional inputs, such as raw images, remains a significant challenge.
- Bootstrapping Stability: Like all TD-based methods, $\gamma$-models can experience instability during training due to the moving target of the bootstrap. More robust training techniques for generative bootstrapping are needed.
- Policy Conditioning: Because the model is conditioned on the current policy, significant policy changes require the model to be updated, which is a standard hurdle in off-policy model-based RL.
Summary updated