AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay

Mingjia Huo, Yao Fu, Bo Chang, Yaqing Wang, Dawei Zhu, Dylan Zhang, Xinyang Yi, Lichan Hong, Ed H. Chi, Hao Zhang, Jiaxi TangarXiv 2026 · arXiv

TL;DR

AtariBench is a new diagnostic benchmark that evaluates how well Vision-Language Models (VLMs) can reason and act in dynamic, fast-paced environments using 20 classic Atari 2600 games. The study reveals a significant "perception-to-action" gap: even top-tier models like GPT-4 and Gemini struggle with basic motion inference, frequently over-rely on stale text history, and fall into repetitive "action loops" in long-context scenarios.

What Problem Does This Paper Address?

While VLMs have become remarkably proficient at describing static images, their ability to function as active agents in dynamic worlds remains a major hurdle. Most current evaluations focus on static scenes or turn-based tasks, which ignores three critical dimensions of intelligence:

  1. Spatiotemporal Physics: Understanding object trajectories, velocities, and distances.
  2. Sequential Decision-Making: Synthesizing a stream of visual data into a coherent strategy.
  3. Visual Grounding: Anchoring decisions in the present visual state rather than relying on historical text or static priors.

The authors argue that to move from passive observers to active agents, VLMs must be tested in environments where timing and motion are paramount.

Methods

The researchers built AtariBench on top of the Arcade Learning Environment (ALE). To make this work for modern VLMs, they introduced a few clever design choices:

  • Decoupled Dynamics: They decoupled in-game speed from model inference time. The environment pauses while the VLM "thinks," but the model must still process high-temporal-resolution streams (10 Hz control) to react to fast-moving objects.
  • The Harness: A standardized agentic harness manages instructions, memory, and action execution.
  • Memory Modes: They compared two ways of providing history to the model:
  • Structured History: A compact buffer of the most recent frames plus "critical" frames where rewards or life-losses occurred.
  • Append-Only: A traditional chat-like transcript containing the full history (which can exceed 100k tokens).
  • Separability Analysis: A rigorous statistical framework to ensure that the chosen games actually distinguish between model capabilities rather than just reflecting random noise.
Figure 1: Workflow of AtariBench. At each turn, the harness assembles game instructions, interac- tion history, and the current frame into a prompt; the VLM returns a thought and an action sequence, which is executed in ALE, and the resulting frames and feedback are added to the history.

Datasets

The benchmark utilizes 20 Atari 2600 games selected based on three principles: they must require motion perception (no card games), be human-interpretable within 30 seconds, and cover diverse genres (Shooters, Sports, Action, and Maze games).

The authors also curated a diagnostic set of 60 manually labeled questions to isolate specific failures in perception, motion comprehension, and decision-making.

Key Insights and Ideas

  • Motion Blindness: Models frequently fail to infer the direction of moving objects. In Breakout, for instance, a model might see the ball but incorrectly guess its trajectory, leading to a missed save.
  • Text-History Overreliance: Models often prioritize old textual feedback over new visual evidence. If a model previously received a "positive score" for firing upward, it might keep firing in that direction even after the enemy has moved away, ignoring the current frame.
  • The Long-Context Trap: In the "append-only" mode, models often suffer from "action stagnation." As the history grows, they tend to collapse into pathological loops, repeating the same action (like noop) regardless of what is happening on screen.
  • Reasoning Doesn't Always Help: Surprisingly, increasing the "thinking" effort or reasoning chain length doesn't consistently improve scores and can sometimes amplify hallucinations or cause the model to ignore visual inputs.
Figure 2: Fast temporal dynamics in Assault. The enemy projectile appears at t = 0.1s and already hits the player at t = 0.3s, motivating high tempo- ral resolution and 10 Hz control.

Key Results

  • VLM vs. RL Gap: While state-of-the-art VLMs (Gemini 3.1 Pro and GPT-5.4) outperformed random baselines, they significantly lagged behind specialized Reinforcement Learning (PPO) agents, which ranked first in 14 out of 16 shared games.
  • Top Performers: Gemini 3.1 Pro and GPT-5.4 occupied the strongest average-rank positions.
  • Separability: The authors' analysis confirmed that 18 of the 20 games are "strongly separable," meaning they provide a clear signal to distinguish model performance.
Figure 3: Model performance ranks across the 20 AtariBench games used in the main analysis, including the standard structured-history settings and the Gemini 3.1 Pro append-only (AO) settings. Game-specific RL agents and a random-action baseline are also included. Rows are ordered by average per-game rank; raw reward values are reported in Appendix B.1.

Limitations and Future Work

  • Fixed Horizon: The evaluation uses a fixed 30-second window. While this reduces variance, it might not capture long-term strategic planning required for some games.
  • VLM-as-a-Judge: The failure attribution relies partly on a VLM judge, which, while accurate (93%), may have its own inherent biases.
  • Cost: Running these evaluations via API is expensive; the authors estimated a cost of ~$522 for 200 runs of Gemini 3.1 Pro.
  • Future Directions: The authors suggest that future VLMs need to be trained specifically on dynamic, interactive environments to bridge the gap between passive perception and active execution.

Summary updated

Abstract

Vision-Language Models (VLMs) have achieved strong performance on static visual understanding benchmarks, yet their ability to act in dynamic environments remains less well understood. We introduce AtariBench, a controlled benchmark that evaluates VLMs across 20 Atari 2600 games, requiring models to make sequential decisions from high-temporal-resolution gameplay streams. By decoupling fast in-game dynamics from wall-clock inference speed, AtariBench focuses on whether models can perceive rapid visual state changes, reason over spatiotemporal dynamics, and ground actions in the current observation. Our experiments reveal substantial limitations in state-of-the-art VLMs: models often fail to infer object motion and direction, exhibit brittle sequential decision-making, and over-rely on textual interaction history despite newly appended visual feedback. We further introduce a separability analysis to identify which in-game scores provide reliable signals of model competence, and use the benchmark to study how memory design and prompting strategies affect performance. Overall, AtariBench exposes a persistent gap between passive visual perception and active decision-making, providing a lightweight and rigorous testbed for developing VLMs that can reason and act in dynamic environments.
BibTeX
@misc{huo2026ataribench,
  author = {Mingjia Huo and Yao Fu and Bo Chang and Yaqing Wang and Dawei Zhu and Dylan Zhang and Xinyang Yi and Lichan Hong and Ed H. Chi and Hao Zhang and Jiaxi Tang},
  title = {{AtariBench: Evaluating Spatiotemporal Reasoning of Vision-Language Models in Atari Gameplay}},
  year = {2026},
  eprint = {2611.08755},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08755},
}
Figure