On Improving Deep Reinforcement Learning for POMDPs

Pengfei Zhu, Xin Li, Pascal Poupart

Abstract

Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.

The Challenge of Partial Observability

Standard Deep Reinforcement Learning (DRL) models, such as the Deep Q-Network (DQN), have achieved remarkable success by assuming the environment is a Markov Decision Process (MDP). In an MDP, the current observation is assumed to contain all the information necessary to make an optimal decision. However, real-world applications are rarely so clean. Most environments are Partially Observable Markov Decision Processes (POMDPs), where sensors may fail, objects may be occluded, or the state space is simply too vast to be captured in a single frame.

Previous attempts to address this, such as the Deep Recurrent Q-Network (DRQN), integrated Long Short-Term Memory (LSTM) layers to provide the agent with a memory of past observations. While an improvement, DRQN overlooks a fundamental principle of POMDP theory: to accurately update a "belief" about the world's state, an agent needs to know both what it saw and what action it took to get there. This paper introduces ADRQN to bridge this gap by explicitly incorporating action histories into the recurrent architecture.

The ADRQN Architecture

The core innovation of the Action-specific Deep Recurrent Q-Network (ADRQN) is the coupling of the agent's previous action with its current observation before the data enters the recurrent memory.

Figure 1: The ADRQN architecture processes action-observation pairs through an LSTM to maintain an accurate belief state over time.

As shown in the architecture diagram above, the process follows five key steps:

  1. Observation Processing: Visual frames are passed through standard Convolutional Neural Network (CNN) layers to extract high-dimensional features.
  2. Action Embedding: To prevent numerical instability when combining a small one-hot action vector with massive image features, the action is passed through a fully connected layer to scale it into a 512-D continuous vector.
  3. Integration: The embedded action and CNN features are concatenated to form a unified "action-observation pair."
  4. Recurrent Memory: This sequence of pairs is fed into an LSTM layer, which integrates the history to infer the underlying hidden state of the environment.
  5. Q-Value Prediction: The LSTM output is processed by a final fully connected layer to determine the optimal next action.

Experimental Evaluation: Flickering Atari

To evaluate ADRQN, the authors utilized the Arcade Learning Environment but modified the games to create "Flickering Atari." In this setup, every frame has a set probability (e.g., $p=0.5$) of being completely obscured by black pixels. This transformation turns standard games into grueling POMDPs where agents must rely on memory to navigate during "blackouts."

Key Results and Robustness

ADRQN demonstrated significant performance gains over both DRQN and its variants. In the game Frostbite with a 0.5 flickering probability, ADRQN achieved a score of 2002.5, vastly outperforming DRQN (673.5) and DDRQN (393). Similar trends were observed in Chopper Command, where ADRQN led with a score of 1608 against DRQN's 1090.

Figure 6: Performance of ADRQN vs. baselines across varying levels of observability in Frostbite. ADRQN maintains high scores even as observability decreases.

Beyond raw performance, ADRQN showed remarkable generalization. As illustrated in the figure above, when models trained at $p=0.5$ were tested across the full spectrum of observability (0.0 to 1.0), ADRQN proved much more robust. It maintained high scores across a wide range of conditions, whereas the performance of baseline models degraded rapidly as the environment became more obscured.

Conclusion

By explicitly modeling the relationship between actions and observations, ADRQN provides a more theoretically sound approach to reinforcement learning in partially observable domains. While the model currently uses truncated backpropagation due to computational limits, its ability to bridge gaps in observation through action-aware memory marks a significant step forward for DRL in complex, real-world environments.

Summary updated

Abstract

Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.
BibTeX
@misc{zhu2017improving,
  author = {Pengfei Zhu and Xin Li and Pascal Poupart},
  title = {{On Improving Deep Reinforcement Learning for POMDPs}},
  year = {2017},
  eprint = {1704.07978},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1704.07978},
}
Figure