Think Offline, Act in Real Time: Self-Improving Visual Game Agents through Code

Zipeng Qiu, Shusheng Xu, Jiale Li, Yukun Zhou, Chenyue Li, Wenshuang Li, Tongkai Yang, Binhang YuanarXiv 2026 · arXiv

TL;DR

This paper demonstrates that for real-time visual games, it is far more effective for an AI to "think offline" and write an executable program than to attempt direct control. By distilling its reasoning into code between game attempts, a 27B model achieved 47.80% mean progress across 170 diverse tasks, significantly outperforming a specialized 2B real-time controller.

What Problem Does This Paper Address?

The "thinking" time of large vision-language models (VLMs) is a major hurdle for real-time applications. In a fast-paced game, the environment doesn't pause while a model processes pixels and decides on an action; by the time the model responds, the game state has already changed.

Furthermore, when an agent encounters an unfamiliar game, it must self-improve from its own failures. Traditional reinforcement learning (RL) requires massive amounts of data and compute to update model weights for a single task. The authors ask: how can an agent efficiently turn a few failed attempts into an effective policy when it only sees the screen and the clock is always ticking?

Methods

The researchers compared two distinct paradigms for self-improvement:

  1. Model as Policy (RTC-2B): Here, a 2B-parameter model acts as the direct controller. To handle inference latency, the authors adapted Real-Time Chunking (RTC). Instead of predicting a single action, the model generates a "chunk" of future actions. While the game executes the current queue, the model "thinks" ahead to generate the next chunk. This model was trained using Supervised Fine-Tuning (SFT) on traces from a larger teacher model, followed by a request-level variant of Group Relative Policy Optimization (GRPO).
  2. Code as Policy: In this approach, a larger model (Qwen3.8-27B) acts as a "developer." It observes a recording of a failed attempt, then writes, tests, and revises a Python program. This program—the "code policy"—is what actually plays the game in real-time. It handles perception (e.g., using OpenCV to find the player) and control logic (e.g., "if player_y > goal_y, press UP").
The two studied approaches: direct model control (left) vs. code development (right).

Datasets

  • GameWorld: The primary benchmark, consisting of 170 tasks across 34 games in five categories: Arcade, Platformer, Puzzle, Runner, and Simulation.
  • js13kgames: A collection of 395 small browser games used to pre-train the RTC-2B controller, ensuring it learned general game-playing concepts before being tested on the unseen GameWorld tasks.

Key Insights and Ideas

  • Code as Distilled Reasoning: The most striking finding is that storing knowledge in a program is better than storing it in model weights. Code allows the model to perform complex reasoning "offline" (between runs) and then execute that logic at millisecond speeds during play.
  • The Latency-Coverage Tradeoff: In direct control (RTC), longer action chunks provide better "coverage" for slow inference but make the agent less responsive to sudden visual changes. Figure 2 illustrates how the model must plan over a horizon that accounts for its own thinking time.
  • One Frame is Enough: Surprisingly, the "developer" model doesn't always need a full video of a failed run to start coding. Starting from just a single initial screenshot yielded nearly identical performance to starting from a full trajectory, as the model learns the game's dynamics through its own subsequent code submissions.
  • Independent Exploration vs. Collaboration: When given a larger "budget" for attempts, having four independent developers try different strategies was more effective than having them peer-review each other's work. Independent runs covered more of the solution space, though peer review was shown to help "unstick" developers who were stuck on wrong assumptions.
The Real-Time Chunking (RTC) mechanism allows the model to overlap thinking with action execution.

Key Results

  • Superiority of Code: The Code as Policy approach reached 47.80% mean progress across all 170 tasks, compared to 30.90% for the RTC-2B controller.
  • Broad Success: Code development led in all five game categories. The gains were most dramatic in "Puzzle" games (70.81% vs 35.74%), where logical reasoning is paramount.
  • Weight Specialization is Slow: Even when a powerful model (GPT-6 Astra) was used to fine-tune the RTC-2B controller's weights for a single game (Stack), the gains were modest compared to what a hand-written or model-generated program could achieve.
  • Human-Level Performance: The best-of-four independent code runs reached 65.21% progress, which is comparable to the 64.1% progress reported for a novice human player on the same benchmark.
The workflow for code development: the model revises its program based on execution feedback.

Limitations and Future Work

The authors identify perception as the primary bottleneck for code policies. While the model can write complex logic, the generated code often struggles to reliably recover the game state (like precise object coordinates) from raw pixels using standard libraries like OpenCV.

Future work could involve providing the developer model with more robust, pre-trained visual primitives or exploring hybrid approaches where a neural network handles perception while a generated program handles high-level strategy. Additionally, the current "developer" model is quite large; finding ways to achieve similar code-generation quality with smaller, more efficient models remains an open challenge.

Summary updated

Abstract

Pre-trained models can read screens and reason about games, yet on an unfamiliar game, an agent still has to learn to play. We ask how an agent can self-improve on such a game from a few of its own attempts, in real-time visual games where the agent sees only the screen and the game keeps running while the agent thinks. We study two kinds of policy that can carry this improvement. In model as policy, a model selects actions during play; we adapt real-time chunking so that committed actions keep executing while the model reasons, and train a 2B controller for this interface. In code as policy, a model writes, tests, and revises a program between attempts, and the program acts during play. Across 170 GameWorld tasks, with at most eight one-minute executions per run, code as policy reaches 47.80% mean progress, against 30.90% for our trained controller, and leads in all five game categories. Improving the controller's weights on a single game, by contrast, yields only modest gains. These results favor letting a model think offline and storing what it learns in a program that acts in real time. Project repository: https://github.com/TabibitoQZP/SIGame
BibTeX
@misc{qiu2026think,
  author = {Zipeng Qiu and Shusheng Xu and Jiale Li and Yukun Zhou and Chenyue Li and Wenshuang Li and Tongkai Yang and Binhang Yuan},
  title = {{Think Offline, Act in Real Time: Self-Improving Visual Game Agents through Code}},
  year = {2026},
  eprint = {2611.24590},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24590},
}
Figure