TL;DR
Researchers from Noetra Corp. have developed a unified scaling law for Vision-Language Models (VLMs) that quantitatively models the trade-offs between "early fusion" (training on mixed modalities from scratch) and "late fusion" (continued pre-training of an LLM on multimodal data). Their findings reveal that while late fusion offers a temporary vision performance boost, this advantage fades as models scale, making early fusion the superior strategy for large-scale foundation models.
What Problem Does This Paper Address?
Training state-of-the-art VLMs is notoriously expensive and often relies on "vibes-based" heuristics. Traditionally, practitioners take a pre-trained LLM and a vision encoder, then perform "late fusion" by training them together on image-text pairs. Recently, "early fusion"—mixing text and multimodal data from the start—has gained popularity, but we lacked a mathematical framework to compare these two paradigms.
Without a unified scaling law, it was impossible to systematically determine which fusion strategy would yield the best performance for a given compute or data budget. This paper fills that gap, providing a way to predict validation losses across different mixture ratios, training steps, and model sizes.
Methods
The authors propose a unified scaling law that decomposes validation loss ($L$) into two distinct components:
- A Base Learning Trajectory: This models how different data types (text, interleaved, or captions) advance progress at different "exchange rates." For example, caption tokens are highly efficient for vision tasks but contribute almost nothing to general text comprehension.
- An Additive Distribution Shift: This term captures the "competition" between modalities. When you suddenly inject dense multimodal data into a text-only model (late fusion), it causes a distribution shift that helps vision but degrades text performance.
To model the mixture ratio ($R$), the authors borrowed a concept from chemical kinetics, treating the model's representational capacity like a catalyst surface where text and multimodal "gases" compete for sites.
Datasets
The study utilized a diverse set of high-quality data:
- Pre-training (PT): FineWeb-Edu for text-only data.
- Continued Pre-training (CPT): OBELICS and OmniCorpus for interleaved image-text data; ReLAION for image-caption pairs.
- Text Benchmarks: HSwag and LAMBADA.
- Vision Benchmarks: MME, GQA, and DocVQA.
A significant contribution is the authors' Rewriting Strategy for evaluation. Standard vision benchmarks usually require instruction-tuned models. To evaluate base (pre-trained only) models fairly, the authors converted multiple-choice questions into declarative statements and used log-likelihood ranking, avoiding the "answer-prior collapse" that often plagues base model evaluation.
Key Insights and Ideas
- The "Exchange Rate" of Data: The law quantifies how much one token type is worth relative to another. Interleaved data is nearly substitutable for text ($w_{int} \approx 1.34$), but caption tokens are essentially useless for text learning ($w_{cap} \approx 0.013$).
- Late Fusion's Diminishing Returns: Late fusion creates a "vision edge" by focusing compute on multimodal data late in the game. However, the authors found that the amplitude of this vision advantage decays as model size ($N$) increases, specifically following a power law of $N^{-0.5}$.
- The Text Penalty is Constant: Unlike the vision gain, the text performance penalty paid during late fusion does not shrink with scale.
- Early Fusion is the Scalable Choice: Because the vision advantage of late fusion disappears at scale while the text penalty remains, early fusion becomes increasingly attractive for larger models.
Key Results
The researchers validated their law across model sizes ranging from 189M to 3.5B parameters.
- Predictive Accuracy: The scaling law accurately predicted validation losses for held-out configurations, including different fusion points and longer training horizons.
- The 1B Scale Shift: Moving from a 300M to a 1B model, the caption loss gap between early and late fusion narrowed by 40%.
- Benchmark Divergence: Text benchmarks closely tracked validation losses. However, vision benchmarks (like MME) did not always improve even when validation caption loss decreased, suggesting that downstream VLM performance requires a delicate balance of both language and vision capabilities that late fusion often disrupts.
Limitations and Future Work
- Heuristic Dependency: As with all scaling laws, the specific constants ($A, B_0, \alpha$) depend on the specific architecture (Gemma-3 style) and data processing choices.
- Language Collapse: The law does not model the "catastrophic collapse" that occurs at extreme mixture ratios ($R \to 1$), where the model completely loses its ability to generate coherent text.
- Scope: The study focused on vision-language; extending this unified framework to other modalities (audio, video) or even larger "frontier" model scales remains a task for future research.
Summary updated