TL;DR
GameRealm is a massive, game-derived dataset featuring over 120 million frames and 1,000 hours of gameplay with rich, synchronized multimodal ground truth. By providing "game-native" supervision—including metric depth, camera poses, and player control inputs—it significantly boosts the performance of visual foundation models in both interactive world modeling and geometric reconstruction tasks.
What Problem Does This Paper Address?
Developing "physical intelligence" in AI requires models to understand not just what the world looks like, but how it is structured geometrically and how it responds to interaction. Currently, researchers face a "data bottleneck":
- Real-world data is difficult and expensive to annotate with precise 3D geometry or camera poses.
- Existing synthetic datasets are often small, lack visual diversity, or rely on noisy estimates rather than true ground-truth parameters.
- Multimodal fragmentation means datasets rarely provide synchronized video, audio, and control inputs (like keyboard/mouse) simultaneously.
GameRealm tackles these issues by leveraging modern video games as high-fidelity simulators that provide direct, "native" access to the underlying engine data.
Methods
The authors developed a unified data collection pipeline designed for scale and quality:
- Hybrid Acquisition: They combine human-operated collection (to capture complex, context-aware interactions like combat or sports) with agentic exploration (where an automated navigation system explores free-roaming environments).
- Game-Engine Plugins: Instead of post-processing videos, they use custom plugins to extract precise metric depth, camera intrinsics/extrinsics, and 3D point trajectories directly from the game engine.
- Quality Control: A multi-stage filtering process uses optical flow to ensure motion consistency and AI-based aesthetic predictors (like DOVER++) to remove visually degraded or "glitchy" clips.
- Model Adaptation: To prove the dataset's utility, the authors fine-tuned several state-of-the-art models (e.g., Helios for video generation, $\pi^3$ for multi-view reconstruction) using Parameter-Efficient Fine-Tuning (LoRA).
Datasets
The primary contribution is the GameRealm Dataset:
- Scale: 120M+ frames, 1,000+ hours across 50+ diverse games.
- Modality: Synchronized RGB-D, camera parameters, audio, keypresses, and 3D point tracking.
- Views: Supports single-view, stereo, multi-view, and 360° panoramic configurations.
The authors also introduce TrajRealm-Bench, a new benchmark for 2D and 3D point tracking across diverse motions and environments.
Key Insights and Ideas
- The "Native" Advantage: The "aha moment" here is that game-native supervision is fundamentally superior to estimated labels. By using the engine's internal math for depth and poses, the models learn from "perfect" geometry, which translates to better performance even on real-world photos.
- Action-Conditioned Dynamics: By synchronizing keyboard/mouse inputs with video, the authors show that models can learn to "play" the game—predicting how a scene will change specifically in response to a user's command (e.g., turning right or moving forward).
- Cross-Domain Generalization: Surprisingly, models trained on stylized game environments (like "ink painting" or "low-poly" styles) generalize remarkably well to realistic real-world benchmarks like KITTI. This suggests that the geometric principles learned in games are universal.
Key Results
- Interactive World Modeling: Fine-tuned models achieved a 76.59% success rate in following directional commands (WASD) during video generation.
- Stereo Reconstruction: Adding GameRealm data reduced errors on 12 out of 13 metrics across real-world benchmarks. For example, it achieved a 14.3% error reduction on the ETH3D benchmark.
- Multi-view Reconstruction: The fine-tuned $\pi^3$ model showed a 34.7% reduction in Absolute Trajectory Error (ATE) on the Sintel dataset.
- Point Tracking: On the new TrajRealm-Bench, fine-tuned trackers improved their 3D localization accuracy (AJ3D) from 25.1% to 40.2%.
Limitations and Future Work
While GameRealm is a massive leap forward, the authors note a few areas for growth:
- Annotation Granularity: Current annotations focus on camera and scene geometry. Future versions aim to include fine-grained data like human/object skeletons and instance-level segmentation masks.
- Physics Gap: While game engines simulate physics, they don't always perfectly match real-world optical phenomena (like complex lens flares or motion blur), which remains a challenge for perfect sim-to-real transfer.
- Long-Horizon Tasks: The current focus is on short-to-medium term visual prediction; extending this to long-term strategic planning in games is a logical next step.
Summary updated