TL;DR
Multi-modal world models often struggle to leverage multiple specialized foundation models simultaneously due to feature conflicts in their shared representations. $M^2$-REPA solves this by decoupling intermediate diffusion features into distinct modality-specific subspaces before aligning them with expert models (like DINOv2 for RGB and DepthAnythingV2 for depth). This approach significantly boosts visual quality and long-term temporal consistency in video generation without adding any computational overhead during inference.
What Problem Does This Paper Address?
World models act as simulators for AI agents, predicting environmental dynamics across multiple modalities such as RGB video, depth maps, and segmentation masks. To improve generation quality, researchers often use Representation Alignment (REPA) to inject structural and semantic priors from large pre-trained foundation models into the early stages of a diffusion model.
However, existing REPA methods are confined to a single modality (usually RGB). When trying to naively align a multi-modal video model with multiple foundation models simultaneously, a "feature conflict" occurs. Because different foundation models (e.g., DINOv2 for RGB and DepthAnythingV2 for depth) capture distinct, specialized priors, forcing a shared diffusion feature space to match all of them at once leads to feature entanglement and training optimization issues, resulting in negligible performance gains.
Methods
The authors introduce $M^2$-REPA (Multi-modal Decoupled Representation Alignment), utilizing a "divide and conquer" strategy to gracefully ingest knowledge from multiple foundation models:
- Modality-Specific Feature Decoupling: Instead of forcing a single shared diffusion feature to match multiple heterogeneous foundation models, lightweight Multi-Layer Perceptron (MLP) layers project the shared intermediate diffusion features into distinct modality-specific subspaces.
- Multi-modal Representation Alignment Loss: Each decoupled feature subspace is aligned with its corresponding "expert" foundation model (DINOv2 for RGB, DepthAnythingV2 for depth, and SAM2 for masks) using a token-wise cosine similarity loss.
- Modality-specific Decoupling Regularization: To ensure these subspaces capture complementary information rather than redundant features, the authors apply a regularization loss based on Centered Kernel Alignment (CKA), minimizing pairwise similarity to encourage orthogonality.
This framework is plug-and-play and is integrated into an autoregressive video diffusion model using Flow Matching and per-frame noise scheduling.
Datasets
The model was trained and evaluated on the following datasets:
- RealEstate10K: Used for camera-pose conditioned real-world scene generation, evaluating short-term (8 frames) and long-term (200 frames) horizons.
- Minecraft: Used for action-conditioned dynamic environment generation across 150-frame sequences.
- OmniWorld-Game: A multi-modal dataset used to evaluate cross-source robustness, ensuring the model generalizes to independently annotated ground truth rather than just mimicking its training teachers.
Key Insights and Ideas
- The "Modality Advantage": Preliminary tests confirmed that foundation models are highly specialized; DINOv2 alignment exclusively guides better RGB generation, while DepthAnythingV2 excels purely at depth generation.
- Feature Decoupling is Essential: Directly forcing shared representations to match heterogeneous foundation models triggers severe feature conflicts. Decoupling them into independent subspaces allows each expert to guide its dedicated modality without interference.
- CKA over Cosine Similarity: For the decoupling regularization, Centered Kernel Alignment (CKA) significantly outperforms standard cosine penalties. CKA's invariance to isotropic scaling and orthogonal transformations makes it uniquely robust at handling variations in feature distributions across different foundation models.
Key Results
- Long-Term Consistency: On RealEstate10K (200 frames), $M^2$-REPA achieved a Fréchet Video Distance (FVD) of 262.50, substantially outperforming the multi-modal baseline (407.50) and single-expert REPA variants (~301–319).
- Mitigating Scene Collapse: Qualitatively, the method prevents "memory drift" and scene collapse during long-horizon generation, keeping objects (like furniture) structurally consistent across hundreds of frames.
- Scalability: The framework successfully scales to large-scale architectures. When applied to TesserAct (a 5-billion parameter model built on CogVideoX-5B), $M^2$-REPA lowered the FVD from 639.3 to 418.3.
Limitations and Future Work
- Dependency on Expert Quality: The robustness of $M^2$-REPA is inherently tied to the quality of the chosen foundation models, meaning any implicit biases within the expert models may transfer to the world model.
- Training Compute: Aligning with multiple experts requires additional training compute to process the foundation backbones, though the authors note this introduces absolutely zero overhead during inference.
- Broader Impact: The dual-use nature of high-fidelity video synthesis underscores an increasing need for robust media authentication tools alongside these generative advancements.
Summary updated