Scaling Laws for Vision-Language Fusion

Seng Pei Liew, Teppei Suzuki, Seiya TokuiarXiv 2026 · arXiv

TL;DR

Researchers from Noetra Corp. have developed a unified scaling law for Vision-Language Models (VLMs) that quantitatively models the trade-offs between "early fusion" (training on mixed modalities from scratch) and "late fusion" (continued pre-training of an LLM on multimodal data). Their findings reveal that while late fusion offers a temporary vision performance boost, this advantage fades as models scale, making early fusion the superior strategy for large-scale foundation models.

What Problem Does This Paper Address?

Training state-of-the-art VLMs is notoriously expensive and often relies on "vibes-based" heuristics. Traditionally, practitioners take a pre-trained LLM and a vision encoder, then perform "late fusion" by training them together on image-text pairs. Recently, "early fusion"—mixing text and multimodal data from the start—has gained popularity, but we lacked a mathematical framework to compare these two paradigms.

Without a unified scaling law, it was impossible to systematically determine which fusion strategy would yield the best performance for a given compute or data budget. This paper fills that gap, providing a way to predict validation losses across different mixture ratios, training steps, and model sizes.

Methods

The authors propose a unified scaling law that decomposes validation loss ($L$) into two distinct components:

  1. A Base Learning Trajectory: This models how different data types (text, interleaved, or captions) advance progress at different "exchange rates." For example, caption tokens are highly efficient for vision tasks but contribute almost nothing to general text comprehension.
  2. An Additive Distribution Shift: This term captures the "competition" between modalities. When you suddenly inject dense multimodal data into a text-only model (late fusion), it causes a distribution shift that helps vision but degrades text performance.

To model the mixture ratio ($R$), the authors borrowed a concept from chemical kinetics, treating the model's representational capacity like a catalyst surface where text and multimodal "gases" compete for sites.

Schematic of early vs. late fusion and the scale-dependent trade-off.

Datasets

The study utilized a diverse set of high-quality data:

  • Pre-training (PT): FineWeb-Edu for text-only data.
  • Continued Pre-training (CPT): OBELICS and OmniCorpus for interleaved image-text data; ReLAION for image-caption pairs.
  • Text Benchmarks: HSwag and LAMBADA.
  • Vision Benchmarks: MME, GQA, and DocVQA.

A significant contribution is the authors' Rewriting Strategy for evaluation. Standard vision benchmarks usually require instruction-tuned models. To evaluate base (pre-trained only) models fairly, the authors converted multiple-choice questions into declarative statements and used log-likelihood ranking, avoiding the "answer-prior collapse" that often plagues base model evaluation.

Figure 3: Validation of the scaling laws on held- out configurations beyond the fitting domain. The law extrapolates to later fusion points and longer CPT horizons. More results in Appendix B.

Key Insights and Ideas

  • The "Exchange Rate" of Data: The law quantifies how much one token type is worth relative to another. Interleaved data is nearly substitutable for text ($w_{int} \approx 1.34$), but caption tokens are essentially useless for text learning ($w_{cap} \approx 0.013$).
  • Late Fusion's Diminishing Returns: Late fusion creates a "vision edge" by focusing compute on multimodal data late in the game. However, the authors found that the amplitude of this vision advantage decays as model size ($N$) increases, specifically following a power law of $N^{-0.5}$.
  • The Text Penalty is Constant: Unlike the vision gain, the text performance penalty paid during late fusion does not shrink with scale.
  • Early Fusion is the Scalable Choice: Because the vision advantage of late fusion disappears at scale while the text penalty remains, early fusion becomes increasingly attractive for larger models.
Fitted shift amplitude as a function of mixture ratio R.

Key Results

The researchers validated their law across model sizes ranging from 189M to 3.5B parameters.

  • Predictive Accuracy: The scaling law accurately predicted validation losses for held-out configurations, including different fusion points and longer training horizons.
  • The 1B Scale Shift: Moving from a 300M to a 1B model, the caption loss gap between early and late fusion narrowed by 40%.
  • Benchmark Divergence: Text benchmarks closely tracked validation losses. However, vision benchmarks (like MME) did not always improve even when validation caption loss decreased, suggesting that downstream VLM performance requires a delicate balance of both language and vision capabilities that late fusion often disrupts.
Scaling laws extended to varying model sizes N.

Limitations and Future Work

  • Heuristic Dependency: As with all scaling laws, the specific constants ($A, B_0, \alpha$) depend on the specific architecture (Gemma-3 style) and data processing choices.
  • Language Collapse: The law does not model the "catastrophic collapse" that occurs at extreme mixture ratios ($R \to 1$), where the model completely loses its ability to generate coherent text.
  • Scope: The study focused on vision-language; extending this unified framework to other modalities (audio, video) or even larger "frontier" model scales remains a task for future research.

Summary updated

Abstract

Vision-language models (VLMs) have emerged as central foundation models combining multimodal perception with text comprehension. While VLMs are traditionally built via late fusion by continued pre-training an off-the-shelf language model on multimodal (image-text) data, mixing modalities from the start of pre-training (early fusion) has recently gained traction. However, the fundamental trade-offs between these paradigms remain poorly understood, resulting in expensive and heuristic-driven training recipes. In this paper, we systematically analyze the learning dynamics of these multimodal pre-training processes, showing that validation losses during vision-language fusion can be modeled by separating the base learning trajectory, where different data types contribute at distinct rates, from a distribution shift driven by the competition between modalities. We derive a scaling law that unifiedly describes these dynamics, enabling accurate loss predictions across mixture ratios, training steps, and model sizes. Crucially, our law characterizes the trade-off between early and late fusion: under a fixed data budget, early fusion preserves text performance while late fusion favors vision performance, yielding a trade-off that increasingly tilts toward early fusion as model size scales. Extensive experiments confirm that our scaling law reliably captures multimodal learning dynamics and governs fusion decisions.
BibTeX
@misc{liew2026scaling,
  author = {Seng Pei Liew and Teppei Suzuki and Seiya Tokui},
  title = {{Scaling Laws for Vision-Language Fusion}},
  year = {2026},
  eprint = {2611.22351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22351},
}
Figure