Future Prediction Can be a Strong Evidence of Good History Representation in Partially Observable Environments

Jeongyeol Kwon, Liu Yang, Robert D. Nowak, Josiah HannaarXiv 2024 · OpenReview

Abstract

Learning good representations of historical contexts is one of the core challenges of reinforcement learning (RL) in partially observable environments. While self-predictive auxiliary tasks have been shown to improve performance in fully observed settings, their role in partial observability remains underexplored. In this empirical study, we examine the effectiveness of self-predictive representation learning via future prediction, i.e., predicting next-step observations as an auxiliary task for learning history representations, especially in environments with long-term dependencies. We test the hypothesis that future prediction alone can produce representations that enable strong RL performance. To evaluate this, we introduce $\texttt{DRL}^2$, an approach that explicitly decouples representation learning from reinforcement learning, and compare this approach to end-to-end training across multiple benchmarks requiring long-term memory. Our findings provide evidence that this hypothesis holds across different network architectures, reinforcing the idea that future prediction performance serves as a reliable indicator of representation quality and contributes to improved RL performance.

Introduction

In real-world applications, agents often operate in Partially Observable Markov Decision Processes (POMDPs), where they lack a complete view of the environment state. To act optimally, agents must maintain a history of past observations. The standard approach is end-to-end (E2E) training, where a single model learns to summarize history and optimize a policy simultaneously. However, reward signals in RL are often sparse and high-variance, making it difficult to learn effective history representations, particularly for tasks with long-term dependencies. This research investigates whether decoupling representation learning from the RL objective by focusing on future observation prediction can yield superior results.

Methodology: The $\texttt{DRL}^2$ Framework

The authors propose $\texttt{DRL}^2$ (Decoupled Representation Learning and Reinforcement Learning), which separates the learning process into two distinct stages:

  1. Predictive State Representation (PSR) Learning: A history summarizer ($\phi$) is trained to predict future observations. This self-supervised task uses a dense signal—the environment's actual transitions—to build a robust summary of the past.
  2. Reinforcement Learning: The history summarizer is frozen, and its outputs are used as fixed representations for a separate policy network ($\pi$) trained via standard RL algorithms.
The $DRL^2$ architecture decoupling representation learning from policy optimization.

By decoupling these phases, the architecture ensures that the representation is not solely dependent on the potentially weak or delayed reward signal.

Experimental Setup

The study evaluates $\texttt{DRL}^2$ against E2E baselines across several challenging benchmarks, including POPGym (memory-intensive games like Minesweeper and Battleship), modified MuJoCo tasks with partial observability, and Credit Assignment Benchmarks where rewards are significantly delayed.

Results and Discussion

The experiments reveal that $\texttt{DRL}^2$ significantly outperforms E2E training in environments requiring long-term memory. For instance, in Battleship (Hard) and Minesweeper (Hard), $\texttt{DRL}^2$ achieved near-optimal returns while E2E methods struggled to make progress.

A key finding is the strong inverse correlation between the auxiliary task error (PSRLoss) and the final RL performance (Return). As shown in the figure below, lower prediction error consistently leads to higher rewards across various environments.

Correlation between PSRLoss and RL Return across different environments.

This supports the hypothesis that future prediction is a reliable proxy for representation quality. While E2E training can be more sample-efficient for short-term memory tasks, the decoupled approach is essential for solving complex, long-horizon POMDPs where the reward signal alone is insufficient to guide representation learning.

Conclusion

This study demonstrates that future prediction is a powerful auxiliary task for learning history representations in POMDPs. By decoupling representation learning from reinforcement learning, the $\texttt{DRL}^2$ framework provides a more stable and effective way to handle long-term dependencies. The results suggest that the quality of future prediction serves as a strong indicator of an agent's ability to succeed in partially observable environments, regardless of the underlying network architecture.

Summary updated

Abstract

Learning good representations of historical contexts is one of the core challenges of reinforcement learning (RL) in partially observable environments. While self-predictive auxiliary tasks have been shown to improve performance in fully observed settings, their role in partial observability remains underexplored. In this empirical study, we examine the effectiveness of self-predictive representation learning via future prediction, i.e., predicting next-step observations as an auxiliary task for learning history representations, especially in environments with long-term dependencies. We test the hypothesis that future prediction alone can produce representations that enable strong RL performance. To evaluate this, we introduce $\texttt{DRL}^2$, an approach that explicitly decouples representation learning from reinforcement learning, and compare this approach to end-to-end training across multiple benchmarks requiring long-term memory. Our findings provide evidence that this hypothesis holds across different network architectures, reinforcing the idea that future prediction performance serves as a reliable indicator of representation quality and contributes to improved RL performance.
BibTeX
@misc{kwon2024future,
  author = {Jeongyeol Kwon and Liu Yang and Robert D. Nowak and Josiah Hanna},
  title = {{Future Prediction Can be a Strong Evidence of Good History Representation in Partially Observable Environments}},
  year = {2024},
  eprint = {2402.07102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.07102},
}
Figure