Abstract
Current Reinforcement Learning (RL) benchmarks, primarily centered on the Arcade Learning Environment (ALE), often fail to capture two critical real-world challenges: the need for abrupt strategy shifts based on latent context and sensitivity to real-world temporal intervals. This paper introduces Space Fortress (SF) as a new benchmark to address these gaps. Originally designed to study human skill acquisition, SF requires agents to manage complex timing and state-dependent rules. Our evaluation shows that while state-of-the-art (SOTA) agents fail under default conditions, they can achieve superhuman performance when context identification is simplified. These findings highlight a major blind spot in current algorithmic design regarding temporal and contextual reasoning.
Introduction
The success of deep Reinforcement Learning has been largely measured against Atari games. While these benchmarks have driven significant progress, they often allow for consistent, frame-by-frame strategies that do not reflect the dynamic nature of real-world tasks. Specifically, most current benchmarks treat "time" merely as a sequence of discrete frames and do not require agents to adapt their behavior based on hidden environmental contexts.
In contrast, real-world applications demand:
- Context-Dependent Strategy Shifts: The ability to switch between vastly different behaviors based on the current state (e.g., transitioning from precision handling to high-speed transport).
- Temporal Sensitivity: The requirement to perform actions within specific real-time windows, independent of the simulation's frame rate.
This paper argues that current RL algorithms are "context-insensitive" and lack a robust conceptualization of time as an independent variable. To bridge this gap, we present Space Fortress as a challenging alternative benchmark.
The Space Fortress Environment
Space Fortress was originally developed in the 1980s by psychologists to investigate how humans learn complex skills. We have adapted this game into an OpenAI Gym environment to serve as a rigorous testbed for RL agents.
The game's primary challenge lies in its "hidden" logic. To destroy the central fortress, a player must first hit it ten times with a minimum interval of 250ms between shots to build "vulnerability." Once the vulnerability counter reaches ten, the player must execute a "rapid double-fire" (two shots within 250ms) to destroy it. Shooting too quickly during the first phase resets the vulnerability counter, creating a "context trap" where the same action (shooting) has different consequences depending on the hidden state of the fortress.
Methodology
We evaluated three SOTA algorithms—Rainbow, PPO, and A2C—using two neural network architectures:
- SF-FF: A standard feed-forward network.
- SF-GRU: A network incorporating a Gated Recurrent Unit (GRU) to provide temporal memory.
To diagnose the specific causes of agent failure, we tested three reward structures:
- Default (Sparse): Rewards are only granted for destroying the fortress or upon the agent's death.
- Dense: Small incremental rewards are given for every successful hit on the fortress.
- AECI (After Easing Context Identification): Rewards are specifically designed to signal changes in the fortress's vulnerability state, effectively "unmasking" the hidden context.
Performance was compared against a human baseline derived from 117 subjects. We tested two versions of the game: "Autoturn" (automatic targeting) and "Youturn" (the full game requiring manual navigation and targeting).
Results and Discussion
Our experiments revealed that standard RL approaches are fundamentally ill-equipped for the challenges posed by Space Fortress. Under the default sparse reward structure, none of the tested algorithms learned to destroy the fortress even once over 45 million training steps.
As shown in the figure above, performance only improved significantly when the "context" was explicitly signaled through the AECI reward structure. With AECI, the PPO agent using the SF-GRU architecture not only learned the game but achieved a score of approximately 2500, surpassing the average human score of 1900. The superiority of the GRU architecture across tests confirms that internal memory is vital for handling the game's temporal constraints.
Furthermore, we found that agents are highly sensitive to the specific timing of the training environment. When the critical time interval (250ms) was altered, agents struggled to generalize, suggesting they learn a specific rhythm tied to frame counts rather than a generalized understanding of time.
Conclusion
Space Fortress exposes a significant limitation in current Reinforcement Learning: the inability to identify and adapt to latent contexts and temporal requirements without explicit guidance. While SOTA agents can achieve superhuman performance when the context is "eased," a truly robust agent must be able to discover these hidden rules through exploration alone. Future research should focus on developing algorithms that can autonomously identify latent contexts and build a more robust, "clock-like" understanding of time to enable better generalization across diverse environments.
Summary updated