Abstract
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models remains opaque. We provide empirical clarity through controlled, from-scratch pretraining experiments, isolating the factors that govern multimodal pretraining without interference from language pretraining. We adopt the Transfusion framework, using next-token prediction for language and diffusion for vision, to train on diverse data including text, video, image-text pairs, and even action-conditioned video. Our experiments yield four key insights: (i) Representation Autoencoder (RAE) provides an optimal unified visual representation by excelling at both visual understanding and generation; (ii) visual and language data are complementary and yield synergy for downstream capabilities; (iii) unified multimodal pretraining leads naturally to world modeling, with capabilities emerging from general training; and (iv) Mixture-of-Experts (MoE) enables efficient and effective multimodal scaling while naturally inducing modality specialization. Through IsoFLOP analysis, we compute scaling laws for both modalities and uncover a scaling asymmetry: vision is significantly more data-hungry than language. We demonstrate that the MoE architecture harmonizes this scaling asymmetry by providing the high model capacity required by language while accommodating the data-intensive nature of vision, paving the way for truly unified multimodal models.
Introduction
While Large Language Models (LLMs) have mastered textual abstractions—the "shadows on the wall" of human knowledge—they often lack a fundamental grasp of the physical world's physics, geometry, and causality. To bridge this gap, the field is moving toward native multimodal models. However, most current approaches "bolt" vision onto a pretrained language backbone, making it impossible to observe how these modalities interact during the core learning phase. This research provides empirical clarity by isolating variables in a controlled, from-scratch environment to determine if vision and language compete for capacity and how to scale them efficiently.
The Transfusion Framework
The study utilizes the Transfusion framework, which treats a single transformer backbone as a multitasker. It performs next-token prediction for discrete text and diffusion (specifically flow matching) for continuous visual latents.
Key methodological choices include:
- Unified Architecture: A decoder-only Transformer using modality-specific Feed-Forward Networks (FFNs). This design choice is critical; empirical results show that modality-specific FFNs significantly outperform shared FFNs across both language and vision metrics.
- Representation Autoencoder (RAE): Instead of using separate encoders for generation (VAE) and understanding (CLIP), the model uses a single RAE based on SigLIP 2. This allows the model to use the same high-dimensional latent space for both tasks, simplifying the architecture without sacrificing performance.
- Mixture-of-Experts (MoE): Dense FFNs are replaced with sparse MoE layers to explore how the model naturally allocates capacity to different modalities.
Data and Scaling Asymmetry
The models were trained on a massive mixture totaling roughly 1 trillion tokens, including DCLM web text, raw video (YouTube, Kinetics), and image-text pairs. A major finding of this work is the Scaling Asymmetry: vision is significantly more data-hungry than language. Through "IsoFLOP" analysis, the authors found that as models grow, the amount of vision data needed increases faster than the amount of text data.
MoE architectures help harmonize this asymmetry. By using many small experts, the model naturally specializes, dedicating more parameters to language (which is parameter-hungry) while efficiently processing the high volume of vision data.
Key Insights and Results
- Cross-Modal Synergy: Vision and language act as teammates rather than competitors. For instance, adding text tokens to a vision-heavy training budget improved image generation quality, suggesting that improved language modeling helps the model better interpret complex visual prompts.
- Emergent World Modeling: World-modeling capabilities—such as predicting the next frame based on an action—emerge primarily from general video pretraining. The model achieved competitive navigation performance with as little as 1% in-domain robotics data and demonstrated zero-shot controllability by following free-form language actions (e.g., "get out of the shadow!").
- Efficiency: MoE architectures halved the gap in scaling exponents between vision and language, achieving a DCLM perplexity of 12.3 and a COCO FID of 39.2 at the $10^{21}$ FLOP scale.
Conclusion and Future Work
This exploration demonstrates that native multimodal pretraining from scratch is not only viable but synergistic. While the current work focuses on discrete blocks of data, future research will look toward interleaved web data and Multimodal Reinforcement Learning (RL) to refine "System 2" reasoning behaviors. Additionally, while RAEs are superior for semantic understanding, closing the "fidelity gap" in fine-grained pixel reconstruction compared to traditional VAEs remains an open challenge.
Summary updated