BLIP-3: A Family of Open Large Multimodal Models

Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S. Ryoo, Shrikant Kendre, Jieyu Zhang, Shao-Yen Tseng, Gustavo A. Lujan-Moreno, Matthew L. Olson, Musashi Hinck, David Cobbley, Vasudev Lal, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming Xiong, Ran Xu

TL;DR

BLIP-3 introduces a comprehensive open-source framework for developing Large Multimodal Models (LMMs), featuring curated datasets, a refined training recipe, and new model architectures. This work aims to democratize LMM research by releasing models, code, and datasets, enabling competitive performance on both single and multi-image tasks, particularly excelling at understanding interleaved image-text inputs.

What Problem Does This Paper Address?

The field of Large Multimodal Models (LMMs) has seen rapid progress, but a significant gap persists between proprietary and open-source models. This gap stems from a lack of transparency in training recipes, limited access to high-quality base models, and insufficient curated datasets in the open-source community. Many existing open-source LMMs don't fully disclose their training details or data, hindering replication and further development. Furthermore, current LMMs often struggle with truly understanding interleaved image-text inputs, which is a natural way humans process information. BLIP-3 aims to bridge this gap by providing a complete, open, and scalable framework for LMM development.

Methods

BLIP-3 builds upon the success of previous models like BLIP-2 but introduces several key improvements:

  • Streamlined Architecture: The intricate Q-Former architecture from BLIP-2 is replaced with a more scalable "vision token sampler" (specifically, a perceiver resampler). This simplifies the model and improves efficiency.
  • Unified Training Objective: Instead of multiple complex losses (ITM, ITC, ITG), BLIP-3 primarily uses a single, auto-regressive next-token prediction loss for multimodal inputs. This simplifies the training process.
  • Any-Resolution Vision Token Sampling: The framework employs a dynamic, "any-resolution" image encoding strategy. This involves splitting images into patches and encoding them separately, allowing the model to process high-resolution images effectively, especially for text-rich content.
  • Stage-by-Stage Training: BLIP-3 follows a two-stage pre-training process:
  • Stage 1 (Base Resolution): Focuses on general image-text alignment using a large mixture of datasets at a base resolution (384x384).
  • Stage 2 (High Resolution): Builds upon Stage 1 with high-resolution datasets and the "any-resolution" sampling strategy to enhance capability with detailed visual information.
  • Instruction Fine-tuning: The pre-trained models are then fine-tuned on a mix of single-image and interleaved multi-image instruction-following datasets to improve their ability to understand and respond to complex multimodal prompts.

Datasets

BLIP-3 leverages a diverse mix of existing and newly created datasets:

  • Newly Introduced Datasets:
    • BLIP3-OCR-200M: A large-scale dataset with 200 million images and dense OCR annotations for general OCR understanding.
  • BLIP3-GROUNDING-50M: A large-scale visual grounding dataset with 50 million images, providing object localization information.
  • BLIP3-OCR-HD-30M: A high-resolution OCR dataset specifically curated for Stage-2 pre-training, featuring high-quality annotations for images with resolutions greater than 512 pixels.
  • Existing Datasets: The framework also incorporates publicly available datasets such as MINT-1T, OBELICS, Datacomp-1B, CC12M, CC3M, VG, SBU, and IDL.
  • Fine-tuning Data: A mixture of approximately 3 million publicly available instruction-following datasets, including both pure text and multimodal image-text data.

Key Insights and Ideas

  • Data is King (and Scale Matters): The paper strongly emphasizes that the scale, quality, and diversity of training data are crucial for robust LMM performance. Scaling up multimodal tokens from 2B to 100B showed significant gains in few-shot performance.
  • Simplification Drives Scalability: Replacing the complex Q-Former with a perceiver resampler and unifying the training objective makes the LMM training process more scalable and efficient.
  • High-Resolution Understanding is Key: The "any-resolution" vision token sampling strategy is vital for tasks requiring detailed visual understanding, especially those involving text within images (OCR).
  • Interleaved Input is Natural: The framework explicitly addresses the need for models to handle interleaved image-text inputs, which is more representative of real-world multimodal interactions than single-image inputs.
  • Openness Fuels Progress: By releasing the entire framework (code, models, datasets), BLIP-3 aims to significantly lower the barrier to entry for LMM research and foster community-driven innovation.

Key Results

BLIP-3 models (4B and 14B) demonstrate competitive performance against other open-source LMMs of similar sizes across various benchmarks.

  • Single-Image Evaluation: The BLIP-3 4B model achieved an average score of 59.2 on single-image benchmarks, outperforming several comparable models. The BLIP-3 14B model achieved an average score of 62.0.
  • Multi-Image Evaluation: The BLIP-3 14B model fine-tuned for interleaved multi-image tasks (BLIP-3-14B-MI) showed significant improvements over its single-image fine-tuned counterpart (BLIP-3-14B-SI), achieving a score of 61.3 compared to 55.3.
  • OCR Performance: Ablation studies showed that Stage-2 high-resolution pre-training significantly improved OCR capabilities.
  • Vision Token Sampler: The perceiver resampler was found to be more effective than an MLP-based approach for vision token sampling, offering a better overall score despite a slightly lower OCR performance.

Limitations and Future Work

  • Computational Cost: While BLIP-3 aims for efficiency, training large multimodal models remains computationally intensive.
  • Instruction-Aware VL Connector: The paper notes that while they experimented with an "instruction-aware" vision token sampling strategy, the difference was not significant enough to warrant its adoption in the main model. Further exploration of this area is suggested for future work.
  • Dataset Curation: While BLIP-3 introduces new datasets, the continuous need for even larger and more diverse datasets for LMMs is implied.
  • Broader Capabilities: Future work could explore expanding the model's capabilities to more complex reasoning tasks and a wider range of modalities. The authors express a hope that their open framework will empower the community to explore these directions.

Summary updated

Abstract

This paper introduces BLIP-3, an open framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. We release 4B and 14B models, including both the pre-trained base model and the instruction fine-tuned ones. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our models demonstrate competitive performance among open-source LMMs with similar model sizes. Our resulting LMMs demonstrate competitive performance among open-source LMMs with similar model sizes, with the ability to comprehend interleaved image-text inputs. Our training code, models, and all datasets used in this work, including the three large-scale datasets we create and the preprocessed ones, will be open-sourced to better support the research community.
BibTeX
@inproceedings{xue2025blip,
  author = {Le Xue and Manli Shu and Anas Awadalla and Jun Wang and An Yan and Senthil Purushwalkam and Honglu Zhou and Viraj Prabhu and Yutong Dai and Michael S. Ryoo and Shrikant Kendre and Jieyu Zhang and Shao-Yen Tseng and Gustavo A. Lujan-Moreno and Matthew L. Olson and Musashi Hinck and David Cobbley and Vasudev Lal and Can Qin and Shu Zhang and Chia-Chih Chen and Ning Yu and Juntao Tan and Tulika Manoj Awalgaonkar and Shelby Heinecke and Huan Wang and Yejin Choi and Ludwig Schmidt and Zeyuan Chen and Silvio Savarese and Juan Carlos Niebles and Caiming Xiong and Ran Xu},
  title = {{BLIP-3: A Family of Open Large Multimodal Models}},
  booktitle = {ICCVW},
  year = {2025},
  url = {https://openreview.net/forum?id=jcqtm2txPl},
}
Figure