GameRealm: Grounding Visual Learning in Interactive Games

Weiguang Zhao, Yifan Wang, Ni Yang, Yating Wang, Xingyu Miao, Yankai Jiang, Mulin Yu, Tao Lu, Haoyu Guo, Junting Dong, Weinan Zhang, Chunhua ShenarXiv 2026 · arXiv

TL;DR

GameRealm is a massive, game-derived dataset featuring over 120 million frames and 1,000 hours of gameplay with rich, synchronized multimodal ground truth. By providing "game-native" supervision—including metric depth, camera poses, and player control inputs—it significantly boosts the performance of visual foundation models in both interactive world modeling and geometric reconstruction tasks.

What Problem Does This Paper Address?

Developing "physical intelligence" in AI requires models to understand not just what the world looks like, but how it is structured geometrically and how it responds to interaction. Currently, researchers face a "data bottleneck":

  1. Real-world data is difficult and expensive to annotate with precise 3D geometry or camera poses.
  2. Existing synthetic datasets are often small, lack visual diversity, or rely on noisy estimates rather than true ground-truth parameters.
  3. Multimodal fragmentation means datasets rarely provide synchronized video, audio, and control inputs (like keyboard/mouse) simultaneously.

GameRealm tackles these issues by leveraging modern video games as high-fidelity simulators that provide direct, "native" access to the underlying engine data.

Methods

The authors developed a unified data collection pipeline designed for scale and quality:

  • Hybrid Acquisition: They combine human-operated collection (to capture complex, context-aware interactions like combat or sports) with agentic exploration (where an automated navigation system explores free-roaming environments).
  • Game-Engine Plugins: Instead of post-processing videos, they use custom plugins to extract precise metric depth, camera intrinsics/extrinsics, and 3D point trajectories directly from the game engine.
  • Quality Control: A multi-stage filtering process uses optical flow to ensure motion consistency and AI-based aesthetic predictors (like DOVER++) to remove visually degraded or "glitchy" clips.
  • Model Adaptation: To prove the dataset's utility, the authors fine-tuned several state-of-the-art models (e.g., Helios for video generation, $\pi^3$ for multi-view reconstruction) using Parameter-Efficient Fine-Tuning (LoRA).
Figure 2. Data acquisition pipeline

Datasets

The primary contribution is the GameRealm Dataset:

  • Scale: 120M+ frames, 1,000+ hours across 50+ diverse games.
  • Modality: Synchronized RGB-D, camera parameters, audio, keypresses, and 3D point tracking.
  • Views: Supports single-view, stereo, multi-view, and 360° panoramic configurations.

The authors also introduce TrajRealm-Bench, a new benchmark for 2D and 3D point tracking across diverse motions and environments.

Figure 1. GameRealm overview. GameRealm spans diverse games and supports multiple tasks: single-view, multi-view, and panoramic world modeling, alongside stereo, multi-view reconstruction, and point tracking. The collage showcases original game ground-truth data totaling over 120 million frames and 1,000 hours.

Key Insights and Ideas

  • The "Native" Advantage: The "aha moment" here is that game-native supervision is fundamentally superior to estimated labels. By using the engine's internal math for depth and poses, the models learn from "perfect" geometry, which translates to better performance even on real-world photos.
  • Action-Conditioned Dynamics: By synchronizing keyboard/mouse inputs with video, the authors show that models can learn to "play" the game—predicting how a scene will change specifically in response to a user's command (e.g., turning right or moving forward).
  • Cross-Domain Generalization: Surprisingly, models trained on stylized game environments (like "ink painting" or "low-poly" styles) generalize remarkably well to realistic real-world benchmarks like KITTI. This suggests that the geometric principles learned in games are universal.

Key Results

  • Interactive World Modeling: Fine-tuned models achieved a 76.59% success rate in following directional commands (WASD) during video generation.
  • Stereo Reconstruction: Adding GameRealm data reduced errors on 12 out of 13 metrics across real-world benchmarks. For example, it achieved a 14.3% error reduction on the ETH3D benchmark.
  • Multi-view Reconstruction: The fine-tuned $\pi^3$ model showed a 34.7% reduction in Absolute Trajectory Error (ATE) on the Sintel dataset.
  • Point Tracking: On the new TrajRealm-Bench, fine-tuned trackers improved their 3D localization accuracy (AJ3D) from 25.1% to 40.2%.
Figure 3. Data collection. GameRealm encompasses diverse game environments, first- and third- person perspectives, and a wide range of playable characters, including humans and animals. Data are acquired through two complementary strategies: human-driven collection for interaction-rich scenarios (top) and automated collection for free-roaming scenarios (bottom)

Limitations and Future Work

While GameRealm is a massive leap forward, the authors note a few areas for growth:

  • Annotation Granularity: Current annotations focus on camera and scene geometry. Future versions aim to include fine-grained data like human/object skeletons and instance-level segmentation masks.
  • Physics Gap: While game engines simulate physics, they don't always perfectly match real-world optical phenomena (like complex lens flares or motion blur), which remains a challenge for perfect sim-to-real transfer.
  • Long-Horizon Tasks: The current focus is on short-to-medium term visual prediction; extending this to long-term strategic planning in games is a logical next step.

Summary updated

Abstract

Visual foundation models are central to physical intelligence, yet their development remains constrained by fragmented training datasets, inconsistent annotation quality, and limited multimodal supervision. To address these challenges, we introduce GameRealm, a large-scale, richly annotated dataset that grounds visual observations in synchronized, game-native supervision. GameRealm encompasses over 120 million RGB frames and 1,000 hours of gameplay footage across all views. Beyond its unprecedented scale, GameRealm provides rich multimodal supervision, including metric depth, camera parameters, synchronized audio, control inputs, and 3D point trajectories. Its diversity further extends to flexible view configurations, spanning single-view, stereo, multi-view, and panoramic recordings. To enable scalable acquisition across diverse game environments, we develop a unified data collection pipeline that combines human-operated interaction with agentic exploration. Extensive experiments across diverse visual learning tasks demonstrate that GameRealm consistently strengthens visual foundation models in both visual generation and geometric learning. These results establish GameRealm as a versatile and valuable data resource, unlocking new opportunities for advancing visual intelligence.
BibTeX
@misc{zhao2026gamerealm,
  author = {Weiguang Zhao and Yifan Wang and Ni Yang and Yating Wang and Xingyu Miao and Yankai Jiang and Mulin Yu and Tao Lu and Haoyu Guo and Junting Dong and Weinan Zhang and Chunhua Shen},
  title = {{GameRealm: Grounding Visual Learning in Interactive Games}},
  year = {2026},
  eprint = {2611.23140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23140},
}
Figure