Abstract
Deep Reinforcement Learning (RL) recently emerged as one of the most competitive approaches for learning in sequential decision making problems with fully observable environments, e.g., computer Go. However, very little work has been done in deep RL to handle partially observable environments. We propose a new architecture called Action-specific Deep Recurrent Q-Network (ADRQN) to enhance learning performance in partially observable domains. Actions are encoded by a fully connected layer and coupled with a convolutional observation to form an action-observation pair. The time series of action-observation pairs are then integrated by an LSTM layer that learns latent states based on which a fully connected layer computes Q-values as in conventional Deep Q-Networks (DQNs). We demonstrate the effectiveness of our new architecture in several partially observable domains, including flickering Atari games.
The Challenge of Partial Observability
Standard Deep Reinforcement Learning (DRL) models, such as the Deep Q-Network (DQN), have achieved remarkable success by assuming the environment is a Markov Decision Process (MDP). In an MDP, the current observation is assumed to contain all the information necessary to make an optimal decision. However, real-world applications are rarely so clean. Most environments are Partially Observable Markov Decision Processes (POMDPs), where sensors may fail, objects may be occluded, or the state space is simply too vast to be captured in a single frame.
Previous attempts to address this, such as the Deep Recurrent Q-Network (DRQN), integrated Long Short-Term Memory (LSTM) layers to provide the agent with a memory of past observations. While an improvement, DRQN overlooks a fundamental principle of POMDP theory: to accurately update a "belief" about the world's state, an agent needs to know both what it saw and what action it took to get there. This paper introduces ADRQN to bridge this gap by explicitly incorporating action histories into the recurrent architecture.
The ADRQN Architecture
The core innovation of the Action-specific Deep Recurrent Q-Network (ADRQN) is the coupling of the agent's previous action with its current observation before the data enters the recurrent memory.
As shown in the architecture diagram above, the process follows five key steps:
- Observation Processing: Visual frames are passed through standard Convolutional Neural Network (CNN) layers to extract high-dimensional features.
- Action Embedding: To prevent numerical instability when combining a small one-hot action vector with massive image features, the action is passed through a fully connected layer to scale it into a 512-D continuous vector.
- Integration: The embedded action and CNN features are concatenated to form a unified "action-observation pair."
- Recurrent Memory: This sequence of pairs is fed into an LSTM layer, which integrates the history to infer the underlying hidden state of the environment.
- Q-Value Prediction: The LSTM output is processed by a final fully connected layer to determine the optimal next action.
Experimental Evaluation: Flickering Atari
To evaluate ADRQN, the authors utilized the Arcade Learning Environment but modified the games to create "Flickering Atari." In this setup, every frame has a set probability (e.g., $p=0.5$) of being completely obscured by black pixels. This transformation turns standard games into grueling POMDPs where agents must rely on memory to navigate during "blackouts."
Key Results and Robustness
ADRQN demonstrated significant performance gains over both DRQN and its variants. In the game Frostbite with a 0.5 flickering probability, ADRQN achieved a score of 2002.5, vastly outperforming DRQN (673.5) and DDRQN (393). Similar trends were observed in Chopper Command, where ADRQN led with a score of 1608 against DRQN's 1090.
Beyond raw performance, ADRQN showed remarkable generalization. As illustrated in the figure above, when models trained at $p=0.5$ were tested across the full spectrum of observability (0.0 to 1.0), ADRQN proved much more robust. It maintained high scores across a wide range of conditions, whereas the performance of baseline models degraded rapidly as the environment became more obscured.
Conclusion
By explicitly modeling the relationship between actions and observations, ADRQN provides a more theoretically sound approach to reinforcement learning in partially observable domains. While the model currently uses truncated backpropagation due to computational limits, its ability to bridge gaps in observation through action-aware memory marks a significant step forward for DRL in complex, real-world environments.
Summary updated