At Human Speed: Deep Reinforcement Learning with Action Delay

Vlad Firoiu, Tina Ju, Josh TenenbaumarXiv 2018 · arXiv

Introduction

Standard deep reinforcement learning (RL) agents often achieve superhuman performance by exploiting near-zero latency, processing frames and acting on the subsequent step. This creates an uneven comparison with human players, who typically face a reaction delay of approximately 250ms. This paper investigates the impact of human-like action delays on RL performance and proposes a Neural Predictive Model to mitigate these effects, enabling agents to compete fairly in fast-paced environments.

Methodology: The Neural Predictive Model

The authors introduce a model designed to "predict the present." By forecasting the environment state $s_{t+d}$ that will occur when a delayed action is executed, the agent can effectively "undo" the latency handicap.

Key Technical Components

  • Residual-style Architecture: The transition model $P(s, a)$ utilizes three sub-networks: a "delta" network ($D$) for continuous features, a "new" network ($N$) for state construction, and a "forget" network ($F$) for interpolation.
  • Iterative Unrolling: The agent takes the current state and the queue of pending actions, then "unrolls" the predictive model for $p$ steps to estimate the future state.
  • Recurrent Core: The predictive networks are integrated with a Gated Recurrent Unit (GRU) to maintain temporal context.
  • V-trace Correction: The authors employ the V-trace off-policy correction algorithm to stabilize training, as multi-step predictions make the policy highly sensitive to parameter updates.

Experimental Benchmarks

Evaluation was conducted on two primary benchmarks:

  1. Atari 2600: Six classic titles (Beamrider, Breakout, Space Invaders, Ms. Pac-Man, Seaquest, and Q*bert) were used to establish how delay impacts standard RL architectures like IMPALA.
  2. Super Smash Bros. Melee (SSBM): A high-speed professional fighting game used to test the predictive model against both in-game AI and a world-class human professional.

Results and Analysis

The Impact of Latency

In Atari environments, performance was found to be inversely correlated with delay. Without predictive modeling, agents lost significant ground even with small 5-frame delays, suggesting that many "superhuman" AI results rely more on twitch-reflexes than on high-level tactical understanding.

Performance degradation across six Atari 2600 games as action delay increases.

Predictive Efficacy

In head-to-head matchups within SSBM, the predictive agent ($d=4, p=4$) dominated non-predictive counterparts ($d=4, p=0$), proving that the model successfully mitigates the handicap of latency. The most effective agents were those where the number of prediction steps ($p$) exactly matched the frames of delay ($d$).

Comparative rewards for predictive vs. non-predictive agents, demonstrating the model's ability to recover performance lost to latency.

Human Professional Evaluation

The team tested their agent against "Professor Pro," a top-ranked professional player. While the predictive agents (constrained by 6-7 frames of delay) did not win the overall set, they were able to take individual games off a world-class human—a feat impossible for non-predictive agents under the same constraints.

Conclusion and Future Work

While the Neural Predictive Model successfully levels the playing field regarding reaction time, several challenges remain. Current models treat opponents as static parts of the environment rather than intelligent agents. Furthermore, the computational overhead of unrolling predictions limits real-time performance at high delays. Future research will explore opponent modeling, variable delay (jitter) robustness, and Actions Per Minute (APM) constraints to further refine the comparison between human and machine intelligence.

Summary updated

BibTeX
@misc{firoiu2018at,
  author = {Vlad Firoiu and Tina Ju and Josh Tenenbaum},
  title = {{At Human Speed: Deep Reinforcement Learning with Action Delay}},
  year = {2018},
  eprint = {1810.07286},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.07286},
}
Figure