Abstract
This report introduces xGen-MM (also known as BLIP-3), a framework for developing Large Multimodal Models (LMMs). The framework comprises meticulously curated datasets, a training recipe, model architectures, and a resulting suite of LMMs. xGen-MM expands the Salesforce xGen initiative on foundation AI models. Our models undergo rigorous evaluation across a range of tasks, including both single and multi-image benchmarks. Our pre-trained base model exhibits strong in-context learning capabilities and the instruction-tuned model demonstrates competitive performance among open-source LMMs with similar model sizes. In addition, we introduce a safety-tuned model with DPO, aiming to mitigate harmful behaviors such as hallucinations and improve safety. We open-source our models, curated large-scale datasets, and our fine-tuning codebase to facilitate further advancements in LMM research.
Introduction
While Large Multimodal Models (LMMs) like GPT-4V or Gemini have shown incredible capabilities, a significant "transparency gap" exists. Most top-tier models are proprietary, and even "open" models often keep their training data mixtures and specific "recipes" secret. This makes it nearly impossible for the broader research community to replicate, analyze, or improve upon them.
BLIP-3 addresses these hurdles by simplifying the model architecture and focusing on the most critical ingredient for success: large-scale, high-quality data. Developed by Salesforce AI Research in collaboration with Intel Labs and the University of Washington, the project aims to bridge the gap between proprietary models and the open-source community by providing not just model weights, but the full "recipe": curated datasets, training code, and architectural insights.
Methods
The authors move away from the intricate designs of the past toward a more streamlined self-attention style architecture. They utilize SigLIP as the vision transformer (ViT) due to its superior performance on OCR tasks and a Perceiver Resampler (referred to as a Scalable Vision Token Sampler) to downsample image embeddings into a fixed number of visual tokens. The models are built on the Phi-3 family of LLMs and trained using a single, simple objective: next-token prediction.
The training follows a rigorous four-stage pipeline designed to scale effectively:
- Stage-1 (Base Resolution): Pre-training on 100B multimodal tokens at 384x384 resolution.
- Stage-2 (High Resolution): Pre-training on 30B tokens using a dynamic "any-resolution" strategy to capture fine details.
- Stage-3 (Single-Image SFT): Supervised Fine-Tuning for instruction following.
- Stage-4 (Multi-Image SFT): Specific tuning to handle interleaved images and text.
Datasets
The paper introduces three massive, specialized datasets to the public to facilitate research:
- BLIP3-OCR-200M: 200 million images with dense OCR annotations for general text recognition.
- BLIP3-GROUNDING-50M: 50 million images for visual grounding (linking text to specific bounding boxes).
- BLIP3-OCR-HD-30M: 30 million high-resolution images with high-quality OCR annotations.
They also leverage existing giants like MINT-1T (a trillion-token interleaved dataset) and OBELICS to ensure the model understands how images and text relate in natural, free-form contexts.
Key Results
BLIP-3 models demonstrate highly competitive performance against other open-source models of similar size across a wide array of benchmarks.
- BLIP-3-4B: Achieved an average score of 59.2 across single-image benchmarks, outperforming Phi-3-vision (58.4) and VILA-1.5-3B (50.5).
- BLIP-3-14B: Scored 62.0, proving competitive with much larger models like Llama-3.2-11B (60.8) and matching the average performance of GPT-4V (62.1) on the evaluated benchmarks.
- Multi-Image Prowess: On the BLINK benchmark, the 4B-MI (Multi-Image) version jumped from a score of 42.9 to 47.1 compared to its single-image counterpart, highlighting the effectiveness of their interleaved SFT stage.
Conclusion
The authors demonstrate that a simpler architecture can outperform more complex ones if the data recipe is meticulously curated. By releasing the 4B and 14B models alongside three massive new datasets and a transparent training recipe, BLIP-3 provides a high-performance blueprint for the next generation of open-source multimodal AI. Future work will continue to explore architectural refinements and safety tuning to mitigate hallucinations and improve long-form reasoning.
Summary updated