Energy-based Predictive Representations for Partially Observed Reinforcement Learning

Tianjun Zhang, Tongzheng Ren, Chenjun Xiao, Wenli Xiao, Joseph E. Gonzalez, Dale Schuurmans, Bo DaiUAI 2023 · OpenReview

Abstract

In real-world applications, handling partial observability is a common requirement for reinforcement learning algorithms, which is not captured by a Markov decision process (MDP). Although partially observable Markov decision processes (POMDPs) have been specifically designed to address this requirement, they present significant computational and statistical challenges in learning and planning. In this work, we introduce the Energy-based Predictive Representation (EPR) to provide a unified approach for designing practical reinforcement learning algorithms in both the MDP and POMDP settings. This framework enables coherent handling of learning, exploration, and planning tasks. The proposed framework leverages a powerful neural energy-based model to extract an adequate representation, allowing for efficient approximation of Q-functions. This representation facilitates the efficient computation of confidence, enabling the implementation of optimism or pessimism in planning when faced with uncertainty. Consequently, it effectively manages the trade-off between exploration and exploitation. Experimental investigations demonstrate that the proposed algorithm achieves state-of-the-art performance in both MDP and POMDP settings.

Introduction

Most Reinforcement Learning (RL) algorithms assume a Markov Decision Process (MDP), where the agent has a perfect view of the environment's state. In the real world, this is rarely true; sensors are noisy, and parts of the world are hidden (e.g., a robot cannot see behind itself). These are Partially Observable MDPs (POMDPs).

POMDPs are notoriously difficult because the "state" is no longer just the current observation, but the entire history of actions and observations. This leads to the "curse of history," where the complexity of making a decision grows exponentially over time. While researchers have tried using Recurrent Neural Networks (RNNs) or Predictive State Representations (PSRs) to manage this, these methods often struggle with principled exploration and efficient planning in continuous spaces.

Methodology: Energy-Based Predictive Representation (EPR)

The authors propose EPR to bridge the gap between theoretical POMDP models and practical deep RL through several key innovations:

  1. Energy-Based Modeling (EBM): Instead of trying to reconstruct a hidden state, EPR uses an EBM to model the conditional probability of the next observation given a finite window of past history and the current action. This avoids compounding errors found in models that track explicit "beliefs."
  2. The Reparameterization Trick: The transition dynamics of a belief-state MDP are factorized using a kernel-based approach. By applying the Random Fourier Feature (RFF) trick to the EBM, the authors derive a finite-dimensional feature vector $\phi(x, a)$.
  3. Linear Value Functions: The optimal Q-function becomes approximately linear in this feature space, making planning computationally tractable.
  4. Principled Exploration: The linear representation allows for Elliptical Confidence Bounds (similar to UCB). A "bonus" is added to rewards in uncertain areas, effectively managing the exploration-exploitation trade-off.
  5. Training: The model is trained using a stable contrastive loss (Maximum Likelihood Estimation) that scales well with deep neural networks.

Experimental Evaluation

EPR was evaluated on several standard and modified robotic simulation benchmarks:

  • MuJoCo (OpenAI Gym): Standard tasks like HalfCheetah, Ant, and Hopper.
  • DeepMind Control Suite (DMC): Tasks including Cheetah Run and Walker Run, focusing on sparse-reward versions.
  • POMDP Variants: Partially observable versions created by masking velocity information.
  • MetaWorld: Vision-based tasks to test high-dimensional image inputs.

Key Results and Discussion

EPR achieves state-of-the-art performance across both standard and partially observed benchmarks.

Performance comparison on MuJoCo benchmarks. EPR consistently outperforms model-based and model-free baselines across various tasks.

As shown in the table above, EPR outperformed strong baselines like SAC and specialized POMDP solvers. For instance, in the "Ant-ET" task, EPR achieved a score of ~4081 compared to SAC's ~2012.

Sparse Reward Mastery

In environments where rewards are rare, EPR's principled exploration proves critical. In the walker-run-sparse task, EPR scored 701.8, while standard SAC and PPO failed completely.

Performance on DeepMind Control Suite tasks. Note the significant advantage of EPR in sparse reward settings (e.g., walker_run_sparse).

Robustness to Partial Observability

When velocities were masked (the POMDP setting), EPR maintained high performance. For example, on HalfCheetah with partial observations, EPR scored significantly higher than competing representation methods like PSR.

Conclusion

EPR provides a robust, unified framework for RL in both MDP and POMDP settings. By bypassing explicit belief states and leveraging linear representations derived from energy-based models, it enables efficient planning and principled exploration. Future work will explore the impact of history window length and delve deeper into the theoretical underpinnings of this energy-based factorization.

Summary updated

Abstract

In real-world applications, handling partial observability is a common requirement for reinforcement learning algorithms, which is not captured by a Markov decision process (MDP). Although partially observable Markov decision processes (POMDPs) have been specifically designed to address this requirement, they present significant computational and statistical challenges in learning and planning. In this work, we introduce the \emph{Energy-based Predictive Representation (EPR)} to provide a unified approach for designing practical reinforcement learning algorithms in both the MDP and POMDP settings. This framework enables coherent handling of \emph{learning, exploration, and planning} tasks. The proposed framework leverages a powerful neural energy-based model to extract an adequate representation, allowing for efficient approximation of Q-functions. This representation facilitates the efficient computation of confidence, enabling the implementation of optimism or pessimism in planning when faced with uncertainty. Consequently, it effectively manages the trade-off between exploration and exploitation. Experimental investigations demonstrate that the proposed algorithm achieves state-of-the-art performance in both MDP and POMDP settings.
BibTeX
@inproceedings{zhang2023energy,
  author = {Tianjun Zhang and Tongzheng Ren and Chenjun Xiao and Wenli Xiao and Joseph E. Gonzalez and Dale Schuurmans and Bo Dai},
  title = {{Energy-based Predictive Representations for Partially Observed Reinforcement Learning}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=Z1BobPmCTy},
}
Figure