xGen-MM (BLIP-3): A Family of Open Large Multimodal Models

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S. Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, Ran Xu

Abstract

This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our pre-trained base model exhibits strong in-context learning capabilities and the instruction-tuned model demonstrates competitive performance among open-source LMMs with similar model sizes. In addition, we introduce a safety-tuned model with DPO, aiming to mitigate harmful behaviors such as hallucinations and improve safety. We open-source our models, curated large-scale datasets, and our fine-tuning codebase to facilitate further advancements in LMM research.

Introduction

While Large Multimodal Models (LMMs) like GPT-4V or Gemini have shown incredible capabilities, a significant "transparency gap" exists. Most top-tier models are proprietary, and even "open" models often keep their training data mixtures and specific "recipes" secret. This makes it nearly impossible for the broader research community to replicate, analyze, or improve upon them.

BLIP-3 addresses these hurdles by simplifying the model architecture and focusing on the most critical ingredient for success: large-scale, high-quality data. Developed by Salesforce AI Research in collaboration with Intel Labs and the University of Washington, the project aims to bridge the gap between proprietary models and the open-source community by providing not just model weights, but the full "recipe": curated datasets, training code, and architectural insights.

Methods

The authors move away from the intricate designs of the past toward a more streamlined self-attention style architecture. They utilize SigLIP as the vision transformer (ViT) due to its superior performance on OCR tasks and a Perceiver Resampler (referred to as a Scalable Vision Token Sampler) to downsample image embeddings into a fixed number of visual tokens. The models are built on the Phi-3 family of LLMs and trained using a single, simple objective: next-token prediction.

Overview of the BLIP-3 framework compared to BLIP-2 and the four-stage training pipeline.

The training follows a rigorous four-stage pipeline designed to scale effectively:

  1. Stage-1 (Base Resolution): Pre-training on 100B multimodal tokens at 384x384 resolution.
  2. Stage-2 (High Resolution): Pre-training on 30B tokens using a dynamic "any-resolution" strategy to capture fine details.
  3. Stage-3 (Single-Image SFT): Supervised Fine-Tuning for instruction following.
  4. Stage-4 (Multi-Image SFT): Specific tuning to handle interleaved images and text.

Datasets

The paper introduces three massive, specialized datasets to the public to facilitate research:

  • BLIP3-OCR-200M: 200 million images with dense OCR annotations for general text recognition.
  • BLIP3-GROUNDING-50M: 50 million images for visual grounding (linking text to specific bounding boxes).
  • BLIP3-OCR-HD-30M: 30 million high-resolution images with high-quality OCR annotations.

They also leverage existing giants like MINT-1T (a trillion-token interleaved dataset) and OBELICS to ensure the model understands how images and text relate in natural, free-form contexts.

Key Results

BLIP-3 models demonstrate highly competitive performance against other open-source models of similar size across a wide array of benchmarks.

Performance comparison of BLIP-3 models against other open-source and proprietary LMMs.
  • BLIP-3-4B: Achieved an average score of 59.2 across single-image benchmarks, outperforming Phi-3-vision (58.4) and VILA-1.5-3B (50.5).
  • BLIP-3-14B: Scored 62.0, proving competitive with much larger models like Llama-3.2-11B (60.8) and matching the average performance of GPT-4V (62.1) on the evaluated benchmarks.
  • Multi-Image Prowess: On the BLINK benchmark, the 4B-MI (Multi-Image) version jumped from a score of 42.9 to 47.1 compared to its single-image counterpart, highlighting the effectiveness of their interleaved SFT stage.

Conclusion

The authors demonstrate that a simpler architecture can outperform more complex ones if the data recipe is meticulously curated. By releasing the 4B and 14B models alongside three massive new datasets and a transparent training recipe, BLIP-3 provides a high-performance blueprint for the next generation of open-source multimodal AI. Future work will continue to explore architectural refinements and safety tuning to mitigate hallucinations and improve long-form reasoning.

Summary updated

Abstract

This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM, short for xGen-MultiModal, expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our pre-trained base model exhibits strong in-context learning capabilities and the instruction-tuned model demonstrates competitive performance among open-source LMMs with similar model sizes. In addition, we introduce a safety-tuned model with DPO, aiming to mitigate harmful behaviors such as hallucinations and improve safety. We open-source our models, curated large-scale datasets, and our fine-tuning codebase to facilitate further advancements in LMM research. Associated resources will be available on our project page above.
BibTeX
@inproceedings{xue2025xgen,
  author = {Le Xue and Manli Shu and Anas Awadalla and Jun Wang and An Yan and Senthil Purushwalkam and Honglu Zhou and Viraj Prabhu and Yutong Dai and Michael S. Ryoo and Shrikant Kendre and Jieyu Zhang and Can Qin and Shu Zhang and Chia-Chih Chen and Ning Yu and Juntao Tan and Tulika Manoj Awalgaonkar and Shelby Heinecke and Huan Wang and Yejin Choi and Ludwig Schmidt and Zeyuan Chen and Silvio Savarese and Juan Carlos Niebles and Caiming Xiong and Ran Xu},
  title = {{xGen-MM (BLIP-3): A Family of Open Large Multimodal Models}},
  booktitle = {ICCVW},
  year = {2025},
  url = {https://openreview.net/forum?id=jcqtm2txPl},
}
Figure