Learning Time-Sensitive Strategies in Space Fortress

Akshat Agarwal, Ryan M. Hope, Katia SycaraarXiv 2018 · DOI · arXiv

TL;DR

This paper introduces "Space Fortress," a novel reinforcement learning (RL) benchmark designed to challenge current deep RL algorithms with time-sensitive gameplay, abrupt strategy shifts, and sparse rewards. The authors demonstrate that state-of-the-art RL methods struggle with this environment and propose modifications to the PPO algorithm, including structural enhancements and reward shaping, to achieve superhuman performance.

What Problem Does This Paper Address?

Deep reinforcement learning has achieved remarkable success in domains like Atari games. However, many real-world scenarios present challenges that are not well-represented in these benchmarks. Specifically, this paper focuses on three critical characteristics:

  • Reward Sparsity: Agents only receive feedback (rewards) infrequently, making it difficult to learn which actions are beneficial.
  • Abrupt Context-Dependent Reversals of Strategy: The optimal course of action can change suddenly based on the current situation, requiring the agent to quickly adapt its behavior.
  • Time-Sensitive Gameplay: The timing of actions is crucial, and deviating from specific temporal constraints can lead to negative consequences.

Existing deep RL algorithms, while powerful, often struggle when faced with these combined challenges. The authors argue that current benchmarks don't adequately capture these complexities, hindering the development of RL agents capable of tackling more realistic, dynamic environments.

Methods

The paper introduces a new RL environment, "Space Fortress," and then proposes modifications to the Proximal Policy Optimization (PPO) algorithm to tackle its challenges.

Space Fortress Environment: This game requires players to fly a ship, destroy a fortress, and avoid enemy fire. Its key features include:

  • Time-Sensitive Firing: Missiles must be fired within a specific rate. Firing too fast before the fortress is vulnerable resets its vulnerability. Firing too slowly after it's vulnerable prevents destruction.
  • Abrupt Strategy Shift: The optimal firing strategy dramatically reverses when the fortress's vulnerability reaches a threshold (10).
  • Sparse Rewards: Significant rewards are given only for destroying the fortress (+100) or being destroyed (-100).

Modified PPO Algorithm: The authors enhance the standard PPO algorithm with several key components:

  • Recurrent Policy Architecture (SF-GRU): A Gated Recurrent Unit (GRU) is incorporated into the policy network. This allows the agent to maintain a memory of past actions and states, crucial for handling the non-Markovian nature of the time-sensitive firing strategy.
  • Reward Shaping: To address reward sparsity and the difficulty of learning the critical timing, intermediate rewards are introduced:
  • A small positive reward (+10) for each increase in the fortress's vulnerability.
  • Clipping of rewards to +1 for positive and -1 for negative outcomes to stabilize training.
  • A bonus (+2) for successfully destroying the fortress.
  • A reduced penalty for firing missiles (-0.05 instead of -2) to differentiate it from ship destruction.
  • Transfer Learning: The authors explore transferring learned policies from a simpler version of the game ("Autoturn," where the ship automatically orients itself) to the more complex version ("Youturn"). This involves initializing the policy network of the target task with weights from the source task.

Datasets

The primary "dataset" used in this paper is the newly introduced Space Fortress environment. This environment is used for both training and evaluating RL agents. The authors also mention human player data collected for comparison, but this is not a dataset in the traditional machine learning sense.

Key Insights and Ideas

  • Current Deep RL is Insufficient for Time-Sensitive, Context-Switching Tasks: The paper convincingly demonstrates that even state-of-the-art RL algorithms struggle with environments that combine sparse rewards, abrupt strategy changes, and strict temporal constraints. This highlights a significant gap in current RL capabilities.
  • Memory is Crucial for Time-Sensitive Strategies: The success of the SF-GRU architecture over the feedforward SF-FF architecture underscores the importance of recurrent memory for learning complex, time-dependent behaviors. The game's non-Markovian nature necessitates remembering past actions.
  • Reward Shaping is Essential, Not Optional: The ablation studies clearly show that without carefully designed reward shaping (intermediate rewards, clipping, and adjusted penalties), agents fail to learn even basic aspects of the game, let alone the critical timing strategy. This emphasizes the difficulty of learning from purely sparse rewards in complex tasks.
  • Transfer Learning Provides a Significant Boost: Transferring knowledge from a simpler version of the game to the more challenging one dramatically improves performance, especially for the recurrent architecture. This suggests that pre-training on related tasks can effectively bootstrap learning in complex environments.
  • Human Performance is a Relevant Benchmark, But RL Can Surpass It: While human players were used as a baseline, the trained RL agents, particularly with PPO and the SF-GRU architecture, achieved significantly higher scores, even outperforming humans who were given instructions. This is a strong indicator of the effectiveness of the proposed methods.

Key Results

  • Superhuman Performance Achieved: The proposed PPO agent with the SF-GRU architecture and modified reward structure achieved superhuman performance on the Space Fortress game, significantly outperforming human players.
  • SF-GRU Outperforms SF-FF: The recurrent SF-GRU architecture consistently outperformed the feedforward SF-FF architecture, achieving higher average rewards and fortress destruction rates.
  • Transfer Learning Improves Performance: PPO-Transfer agents (both SF-FF and SF-GRU) showed substantial performance gains on the "Youturn" game when initialized with weights from the "Autoturn" game. The SF-GRU transfer agent achieved the best overall performance.
  • Ablation Studies Confirm Necessity of Modifications: Removing any single reward modification (no fortress bonus, no vulnerability reward, or original missile penalty) resulted in drastically degraded performance, demonstrating that all three modifications are essential for learning.
  • A3C Struggles: The Asynchronous Advantage Actor-Critic (A3C) algorithm, even with extensive hyperparameter tuning, was unable to match human performance, highlighting the specific challenges of Space Fortress.

Limitations and Future Work

Limitations:

  • Dependence on Reward Shaping: The paper acknowledges that the current success relies heavily on the proposed reward modifications. A key limitation is that the agents still require these modifications to learn effectively.
  • "Autoturn" vs. "Youturn" Performance Gap: While transfer learning helps, there's still a notable performance difference between the simpler "Autoturn" and the more complex "Youturn" versions, even for the best RL agents.

Future Work:

  • Learning Without Reward Modifications: The primary future direction is to train agents to play Space Fortress without the intermediate rewards and bonuses. This would require learning from extremely sparse rewards, a significant challenge for current RL.
  • Learning Critical Points Without Bonuses: Specifically, learning the critical point of fortress vulnerability without the explicit fortress destruction bonus is highlighted as a challenging but important area for future research, crucial for deploying RL in real-world scenarios.
  • Further Exploration of Transfer Learning: Investigating more sophisticated transfer learning techniques could further improve performance and generalization.

Summary updated

Abstract

Although there has been remarkable progress and impressive performance on reinforcement learning (RL) on Atari games, there are many problems with challenging characteristics that have not yet been explored in Deep Learning for RL. These include reward sparsity, abrupt context-dependent reversals of strategy and time-sensitive game play. In this paper, we present Space Fortress, a game that incorporates all these characteristics and experimentally show that the presence of any of these renders state of the art Deep RL algorithms incapable of learning. Then, we present our enhancements to an existing algorithm and show big performance increases through each enhancement through an ablation study. We discuss how each of these enhancements was able to help and also argue that appropriate transfer learning boosts performance.
BibTeX
@misc{agarwal2018learning,
  author = {Akshat Agarwal and Ryan M. Hope and Katia Sycara},
  title = {{Learning Time-Sensitive Strategies in Space Fortress}},
  year = {2018},
  eprint = {1805.06824},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.06824},
}
Figure