Introduction
In many real-world Reinforcement Learning (RL) scenarios, agents operate under partial observability (POMDPs), where they only receive incomplete information about the environment. To act optimally, agents must rely on their history—the sequence of past observations and actions. A popular technique to accelerate training in these settings is "asymmetric actor-critic," which provides the critic with privileged access to the true environment state during training, while the actor remains restricted to observable history for deployment.
However, this paper identifies a critical theoretical flaw in existing asymmetric methods: using the state alone to train the critic introduces mathematical bias and instability. The authors propose an Unbiased Asymmetric Actor-Critic framework that leverages a history-state critic to resolve these issues.
The Theoretical Flaw: History Aliasing
The core problem with traditional asymmetric methods is "history aliasing." In a POMDP, two different histories can lead to the same physical state, but the agent's future behavior might differ based on what it has learned. For example, an agent that has found a key will behave differently at a door than one that hasn't, even if they are at the same location.
A critic that only sees the state $V(s)$ cannot distinguish between these histories, leading to an ill-defined value function. This bias confuses the actor and can cause catastrophic performance collapse in memory-dependent tasks.
Methodology: The History-State Critic
To address this, the authors introduce the history-state critic, $V(h, s)$. In this architecture, the actor continues to use only the history $h$ for online execution. During training, however, the critic receives both the history $h$ and the privileged state $s$.
The authors mathematically prove the "Asymmetric Policy Gradient Theorem," showing that while a state-only critic is biased, the history-state critic is strictly unbiased. The history informs the critic about the agent's current "belief" or intent, while the state provides perfect environmental context, reducing variance without corrupting the gradient.
Experimental Evaluation
The method was evaluated on eight procedurally generated navigation tasks requiring information-gathering and memorization, such as Heaven-Hell, Shopping, and Memory-Four-Rooms. These environments are specifically designed so that a single observation is insufficient to determine the optimal action.
Key Results
- Superior Convergence: The proposed method (
A2C-asym-hs) consistently outperformed both standard symmetric A2C and biased state-critic A2C (A2C-asym-s) across all benchmarks. - Solving Memory-Intensive Tasks: In the Heaven-Hell environment—where an agent must visit a priest to learn which exit is safe—the unbiased history-state critic was the only method capable of solving the task. Biased baselines failed because they could not distinguish between an agent that knew the exit location and one that did not.
- Stability: In the Cleaner domain, the biased baseline suffered from performance collapse after 2 million steps, whereas the unbiased method remained stable throughout training.
Conclusion and Future Work
This research demonstrates that the common practice of using state-only critics in asymmetric training is theoretically unsound for partially observable tasks. By incorporating both history and state into the critic, the proposed method provides a mathematically rigorous and empirically superior alternative. Future work includes extending this theory to value-based methods like DQN and applying it to Multi-Agent RL (MARL), where asymmetric training is already a standard but potentially biased paradigm.
Summary updated