Interleaved Multimodal Pretraining for Vision Language Models

55 concepts35 papers2 proposals

Early vision-language models relied almost entirely on isolated image-caption pairs. In this traditional paradigm, a model looks at a single image and learns to align it with a short, descriptive string of text. While effective for basic mapping, this approach treats every image as an independent island.

The real world, however, is not a disconnected gallery of captioned images. Web pages, textbooks, scientific articles, and medical reports present information as an interleaved stream where multiple images and text blocks are woven together. Shifting from isolated pairs to interleaved multimodal pretraining fundamentally changes how models reason, unlocking capabilities like multi-image comparison, long-form document understanding, and few-shot in-context learning.


Why Image-Caption Pairs Limit Reasoning

To understand why interleaved pretraining works, we must first look at why isolated image-caption pairs fail. When a model is trained exclusively on single images paired with short descriptions, it learns a static, one-to-one mapping. The model never has to maintain context across a sequence, recognize patterns across multiple images, or understand how a visual element modifies the meaning of subsequent text.

As Alayrac et al. demonstrated with Flamingo, real-world web data requires a model to resolve complex dependencies across arbitrary sequences of text and images. When a model is trained on naturally interleaved documents—such as the massive corpora introduced in Multimodal C4 by Awadalla et al. or OBELICS by Laurençon et al.—it is forced to treat images not as isolated inputs, but as contextual tokens within a larger narrative flow.


Unlocking In-Context Learning

The most profound emergent property of interleaved pretraining is multimodal in-context learning. This is the ability of a model to learn a new task at inference time simply by looking at a few examples provided in the prompt, without any weight updates.

Isolated Pair Training:
[Image 1] -> "A golden retriever puppy"
[Image 2] -> "A red sports car"

Interleaved Sequence Training (Few-Shot Simulation):
[Image A] -> Label: Canine | [Image B] -> Label: Vehicle | [Image C] -> Label: ?

By structuring the training data as an interleaved sequence, the model learns the meta-task of comparing and mapping visual concepts to textual tokens on the fly. Derakhshani et al. show in Small Visual Language Models can also be Open-Ended Few-Shot Learners that exposing models to sequences of multiple examples teaches them the mechanism of pattern recognition within a single input window.

When a model is starved of this sequential structure, its few-shot capabilities plateau. In their work on the mOSCAR corpus, Zebaze et al. observed that models trained purely on image-caption pairs fail to improve when given more examples in a prompt. Interleaved document-level pretraining provides the structural scaffolding necessary for the model to scale its performance dynamically with the length of the context.


Preserving Structural and Positional Context

When tasks require reasoning over multiple images, a naive workaround might involve stitching several images into a single grid or collage. However, this approach destroys the sequential and logical flow inherent in structured information.

Interleaving solves this by preserving the exact relative order and narrative positioning of every element. In MV-MATH, Yin demonstrates that maintaining the original structural and positional context of images within text is significantly more effective than merging images into a collage. For instance, in mathematical proofs or scientific explanations, a diagram must appear precisely between specific steps of a derivation to make sense. As Han et al. show with InfiMM-WebMath-40B, this precise spatial-textual alignment allows the model to map mathematical equations directly to their corresponding visual components.

Stitched Collage Approach (Context Lost):
[ Image 1 | Image 2 ] -> Text explains Step 1, then Step 2. (Model must guess which image matches which step)

Interleaved Approach (Context Preserved):
Text explains Step 1 -> [Image 1] -> Text explains Step 2 -> [Image 2]

This sequential preservation is equally critical in specialized domains like medicine. Yang et al. demonstrate in Medical Large Vision Language Models with Multi-Image Visual Ability that sequential interleaved formatting allows models to perform longitudinal reasoning—such as tracking the progression of a disease by comparing a series of patient scans ordered chronologically within a clinical report.


The Data Efficiency of Rich Context

Beyond unlocking novel capabilities, interleaved pretraining is remarkably data-efficient. In an isolated pair, the text caption is often sparse and descriptive. In an interleaved web document, the text surrounding an image contains rich narrative context, logical arguments, and stylistic cues.

The authors of OBELICS found that training on full documents allows models to achieve comparable or superior performance while using roughly ten times fewer images than traditional image-caption setups. The surrounding text provides a dense supervisory signal that helps the model capture higher-level semantic abstractions.

Additionally, this format protects the underlying language model. Tao et al. note in VILA that pretraining on interleaved data maintains the language model's original text distribution and native reasoning capabilities. It prevents the linguistic degradation and "catastrophic forgetting" that frequently occurs when a powerful text model is fine-tuned exclusively on short, fragmented image captions.

Key Foundational Work

Flamingo: a Visual Language Model for Few-Shot Learning (2022) — contributed 3 concept elements to this direction.
Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Text (2023) — contributed 3 concept elements to this direction.
Language Is Not All You Need: Aligning Perception with Language Models (2023) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_interleaved_multimodal_pre_training_6666fe45,
  title = {interleaved multimodal pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interleaved-multimodal-pre-training/rdr_6666fe459cd643f8b4a0a35429e4f83f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_interleaved_multimodal_pre_training_6666fe45_v1,
  title = {interleaved multimodal pre-training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/interleaved-multimodal-pre-training/rdr_6666fe459cd643f8b4a0a35429e4f83f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake