Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent

Run Luo, Kuan Zhang, Qi Deng, Yukun Chen, Zhihao Yang, Yufei Zhang, Renrong Hu, Yi Zhang, Guiyu Ma, Tianyu Fan, Longze Chen, Hao Sun, Jiaming Li, Jiaming Yang, Xiaohao Liu, Tao Tan, Minzheng Wang, Hangtian Jia, Shuai Yuan, Lvdi Wang, Chao Ma, Yuting Zou, Yinhe Zheng, Xiaobo Xia, Dawei Jiang, Shaoqing RenarXiv 2026 · arXiv

Foundation models are moving beyond static text and images, evolving into interactive agents capable of navigating complex environments. While significant progress has been made in 2D games and web interfaces, mastering modern 3D open-world games remains a "boss level" challenge due to their visual complexity, long-horizon tasks, and the need for real-time responsiveness.

The miHoYo Honkai AI R&D Team recently introduced Gamer, a generalist Vision-Language-Action (VLA) framework designed to tackle these challenges head-on. By operating through a non-invasive "screen-to-action" interface, Gamer learns to play diverse games just like a human—perceiving raw pixels and executing native keyboard and mouse commands.

TL;DR

Gamer is a generalist VLA foundation model that masters complex 3D games using only raw pixels and standard keyboard/mouse inputs. By combining a scalable multi-stage training pipeline with a novel real-time inference engine, it bridges the gap between passive visual understanding and sustained, proactive interaction in dynamic, long-horizon environments.

What Problem Does This Paper Address?

Most existing game agents rely on "cheats"—privileged internal game states or custom APIs—that fail to generalize to new titles. Furthermore, standard multimodal models struggle with the "long-horizon" nature of modern games (where quests can last hours) and the strict latency requirements of real-time play.

Mastering such environments requires a delicate coordination of perception, reasoning, memory, and control. Gamer addresses these gaps by providing a unified framework that generalizes across visually and mechanically diverse games without requiring game-specific modifications.

Gamer operates across a broad range of visually and mechanically diverse games via a single screen-to-action interface.

Methods

The authors developed a three-stage training recipe coupled with a high-performance inference backend:

  1. Continued Pre-training (CPT): The model is trained on tens of thousands of hours of gameplay to learn a "transferable control prior." To prevent the model from simply memorizing specific keybinds, they use random key remapping. They also employ repeat-aware loss decay to focus learning on action transitions (like starting to move or turning) rather than long, repetitive button holds. CPT maps synchronized gameplay observations and device events to a shared action representation.

  2. Supervised Fine-Tuning (SFT): This stage teaches the model to follow explicit task objectives. It introduces sparse-thinking, where the model only generates reasoning tokens at critical decision points, and context memory compaction to summarize long interaction histories into a persistent memory. SFT integrates visual observations, task memory, and optional reasoning into executable actions.

  3. Reinforcement Learning (RL): Using a novel algorithm called Temporal-Aware Group Relative Policy Optimization (TA-GRPO), the model is optimized through online feedback. This method addresses sparse rewards and accumulated errors over long trajectories without needing a complex "value critic" model.

  4. Streaming-vLLM: To enable real-time play, the team built a custom inference engine that reuses KV-caches across turns and uses a fast Finite-State Machine (FSM) to "draft" predictable action tokens, drastically reducing latency.
CPT maps synchronized gameplay observations and device events to a shared action representation.
SFT integrates visual observations, task memory, and optional reasoning into executable actions.

Datasets

  • CPT Data: A massive mixture of ~2.4K hours of gameplay from titles like Minecraft, Honkai: Star Rail (HSR), and Dota 2, alongside 6.3B tokens of general multimodal data.
  • SFT Data: ~1K hours of curated HSR trajectories enriched with multi-level instructions (local operations, subgoals, and complete quests).
  • Evaluation Suite: In addition to public benchmarks like MCU and ViZDoom, the authors introduced a new Honkai: Star Rail Evaluation Suite, featuring long-horizon side quests and 40 distinct minigame types to test generalization and multi-stage reasoning.

Key Insights and Ideas

  • Real-time is a Training Requirement: The authors discovered that high inference latency doesn't just make the agent play poorly; it actually degrades online RL. Delayed actions lead to "stale" feedback, which undermines policy updates.
  • Action Scaling is Viable: Even repetitive keyboard and mouse trajectories contain enough signal to support substantial gains when scaling data and learning algorithms.
  • Sparse Reasoning is Efficient: By only "thinking" when necessary (e.g., when a plan fails or a new objective appears), Gamer reduces redundant computation while preserving the reasoning needed for complex decisions.
  • Unified Action Space is the Path to Generalization: Treating keyboard/mouse events as a unified vocabulary allows the model to transfer skills across games that look different but share similar control logic.

Key Results

Gamer demonstrates solid generalization and operational efficiency comparable to skilled human players.

  • Long-Horizon Success: In Honkai: Star Rail side quests, Gamer-RL achieved an 83.6% success rate, a massive improvement over the 13.9% success rate of the base Qwen3-VL model.
  • Inference Speedup: The Streaming-vLLM engine achieved a 20.1x speedup on complex Mixture-of-Experts (MoE) models, bringing end-to-end latency down to just 65ms.
  • Benchmark Performance: Gamer achieved competitive results on a range of conventional 2D and 3D game benchmarks, outperforming many specialized agents.
A comparison of Gamer against other representative game agents highlights its unique combination of real-time execution and diverse training.

Limitations and Future Work

While Gamer is a powerful generalist, the authors acknowledge several areas for growth:

  • Reasoning Bottlenecks: In environments demanding extremely deep deliberation, the current "sparse-thinking" approach might still hit a wall.
  • Resource Constraints: While Streaming-vLLM is efficient, running these large models still requires significant GPU power. Future work will explore model compression for wider deployment.
  • Environment Coverage: The current evaluation focuses on a specific subset of games. Expanding to even more diverse genres and embodied physical tasks is a key next step.

Summary updated

Abstract

We introduce Gamer, a generalist agent framework built for long-horizon tasks in complex interactive environments. Its end-to-end architecture combines perception, memory management, and multi-step reasoning. A shared training and deployment pipeline supports interaction with both conventional 2D graphical interfaces and complex 3D simulated worlds. We first continue pre-training on a dataset constructed from tens of thousands of hours of device-level keyboard and mouse interaction. These trajectories span simulated games and embodied environments, allowing the model to learn environmental dynamics across both physical and virtual domains. Sparse-thinking instruction tuning then substantially reduces redundant computation while preserving the reasoning needed for complex decisions. We train the policy for long-horizon interaction with a stable multi-turn reinforcement learning method that addresses finite context windows, accumulated rollout errors, and sparse rewards. For real-time interaction, a cloud-edge inference engine serves numerous simulated game environments concurrently, supports unbounded-context execution, and keeps online inference latency low. Across multiple challenging 3D open-world games, Gamer demonstrates solid generalization and operational efficiency comparable to skilled players. It also achieves competitive results on a range of conventional 2D game benchmarks. These results suggest that even repetitive keyboard and mouse trajectories can support substantial gains from scaling data and learning algorithms. Training on these trajectories helps Gamer learn complex skills more steadily in environments with both physical and digital properties. This approach offers a path toward generalist agents that can interact across digital and physical worlds.
BibTeX
@misc{luo2026gamer,
  author = {Run Luo and Kuan Zhang and Qi Deng and Yukun Chen and Zhihao Yang and Yufei Zhang and Renrong Hu and Yi Zhang and Guiyu Ma and Tianyu Fan and Longze Chen and Hao Sun and Jiaming Li and Jiaming Yang and Xiaohao Liu and Tao Tan and Minzheng Wang and Hangtian Jia and Shuai Yuan and Lvdi Wang and Xiao Liu and Chao Ma and Yuting Zou and Yinhe Zheng and Xiaobo Xia and Dawei Jiang and Shaoqing Ren},
  title = {{Gamer: A Generalist Vision-Language-Action Foundation Model for Long-Horizon Real-Time Game Agent}},
  year = {2026},
  eprint = {2611.18397},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18397},
}
Figure