Abstract
This report details the winning approach by Team Revontuli for the 1X World Model Challenge. The challenge tasks participants with developing predictive models for humanoid robots across two tracks: pixel-level video generation (Sampling) and latent-space token prediction (Compression). Our solution leverages a massive internet-scale video foundation model (Wan-2.2) adapted via LoRA and state-conditioning for the Sampling track, and a custom-built, memory-efficient Spatio-Temporal Transformer for the Compression track. We achieved first place in both tracks, demonstrating the efficacy of foundation model adaptation and specialized architectural design for robotic world modeling.
Introduction
To operate effectively in complex real-world environments, robots require "world models"—internal simulators that predict the future consequences of actions. Designing architectures that are expressive enough to capture high-dimensional physics while remaining computationally tractable is a significant hurdle in robotics. The 1X World Model Challenge provides a benchmark using real-world humanoid data to evaluate these capabilities in two distinct domains: visual observation prediction (Sampling) and discrete latent code prediction (Compression).
Methods
Sampling Track: Pixel-Space Prediction
For the Sampling track, we adapted Wan-2.2 TI2V-5B, a 5-billion parameter flow-matching video generation model. Rather than training from scratch, we repurposed this foundation model to handle robotic context.
- Video-to-Video Adaptation: We modified the masking strategy to condition generation on multiple past frames rather than a single image.
- State Conditioning: Robot physical states (angles, velocities) were injected into the Diffusion Transformer (DiT) blocks using AdaLN-Zero modulation.
- Efficient Fine-Tuning: We utilized Low-Rank Adaptation (LoRA) with a rank of 32, allowing for memory-efficient training of the massive backbone.
Compression Track: Latent-Space Prediction
For the Compression track, we developed a Spatio-Temporal (ST) Transformer from scratch to predict future discrete latent codes.
- Factored Attention: To avoid the quadratic scaling of standard 3D attention, our ST-Transformer alternates between spatial attention (intra-frame) and temporal attention (inter-frame), ensuring memory costs scale linearly with frame count.
- State Injection: Robot states were processed through MLPs and 1D convolutions before being added as embeddings to the video tokens.
Datasets and Evaluation
Models were trained on the 1X World Model Challenge dataset, featuring real-world humanoid interaction data.
- Sampling Track: Raw video clips (77 frames, downsampled by 4x) were used. Models predicted 16 future frames given 5 conditioning frames.
- Compression Track: Data was processed by the Cosmos 8x8x8 tokenizer (~1.88 billion tokens). Models observed 3 past token grids to predict the next 3.
Key Insights
- Foundation Model Utility: Internet-scale models like Wan-2.2 possess deep physical priors. They can be rapidly adapted for robotics with minimal fine-tuning and appropriate state-conditioning.
- Metric Optimization vs. Realism: In the Sampling track, we found that ensembling predictions to create "blur" in high-motion areas improved PSNR scores, as the metric heavily penalizes sharp but misaligned predictions.
- Decoding Strategy: For discrete latent prediction, simple greedy autoregressive decoding proved highly effective, balancing speed and accuracy.
Results
Team Revontuli secured 1st place in both challenges:
- Sampling Track: Achieved 23.00 dB PSNR. Training took only 36 hours on a B200 cluster, significantly faster than competitors.
- Compression Track: Achieved a Top-500 Cross-Entropy loss of 6.64, outperforming the runner-up (7.50). Training completed in under 17 hours.
Discussion and Limitations
While ensembling for blurriness optimizes PSNR, it may not be ideal for downstream robotic control where sharp trajectories are preferred. Additionally, while the model retains cross-attention layers, text conditioning was not utilized due to lack of descriptions in the dataset. Future work will focus on closing the "autoregressive gap" between teacher-forced training and inference.
Summary updated