MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, Sam Dodge, Keen You, Zhen Yang, Aleksei Timofeev, Mingze Xu, Hong-You Chen, Jean-Philippe Fauconnier, Zhengfeng Lai, Haoxuan You, Zirui Wang, et al.

Introduction

Multimodal Large Language Models (MLLMs) have advanced rapidly, yet smaller-scale models often struggle with specialized tasks such as text-rich image understanding (OCR), precise visual grounding, and multi-image reasoning. Apple’s MM1.5 family (ranging from 1B to 30B parameters) addresses these gaps through a data-centric approach. By focusing on the training lifecycle and data engineering rather than architectural shifts, MM1.5 achieves state-of-the-art performance for its size.

Methods and Architecture

MM1.5 builds upon the foundational architecture of MM1, utilizing a ViT-L/14 vision encoder and a C-Abstractor connector. However, it introduces significant technical innovations to handle high-resolution inputs and spatial reasoning.

Dynamic High-Resolution and Grounding

To support resolutions up to 4 Megapixels (e.g., 2016x2016), MM1.5 employs a dynamic "AnyRes" approach. Instead of static resizing, which loses fine-grained detail, the model dynamically splits images into sub-tiles. Furthermore, the model incorporates specialized coordinate tokens to interpret and generate bounding boxes, enabling native visual referring and grounding capabilities.

The MM1.5 architecture featuring dynamic image splitting for high-resolution support and coordinate tokens for visual grounding.

Three-Stage Training Recipe

The core of MM1.5's performance lies in its refined three-stage training pipeline:

  1. Large-scale Pre-training: Foundational training at 378x378 resolution using image-text pairs and interleaved documents.
  2. Continual Pre-training (OCR Boost): A high-resolution stage dedicated to OCR-rich images and high-quality synthetic captions, essential for fine-grained text recognition.
  3. Supervised Fine-Tuning (SFT): An optimized mixture of general, text-rich, knowledge-based, grounding, and multi-image data.
The three-stage training lifecycle of MM1.5, highlighting the impact of specific data categories at each stage.

Data Engineering and Curation

The researchers utilized a massive array of datasets to study the impact of different data mixtures. This included 2B image-text pairs and 2T text-only tokens for pre-training, alongside an optimized SFT mixture. Notably, the team introduced in-house datasets like COCO-instruct-interleaved to enhance multi-image and in-context learning.

Key Insights

  • The OCR Stage: Dedicated high-resolution "continual pre-training" is significantly more effective for OCR tasks than simply adding OCR data during the final SFT phase.
  • Resolution and Performance: Increasing effective resolution from 0.45MP to 4MP led to substantial gains in document and infographic analysis (e.g., DocVQA).
  • Grounding Trade-offs: While adding grounding data improves spatial awareness, it can slightly degrade general VQA performance. The study identified an optimal data ratio to balance these capabilities.

Experimental Results

MM1.5 demonstrates state-of-the-art results across 35+ benchmarks. The 1B parameter model outperforms competitors like SPHINX-Tiny, while the 30B model shows massive improvements over its predecessor, MM1, particularly in MathVista (from 39.4 to 55.6) and DocVQA (from 75.8 to 91.4). Specialized variants for Video and Mobile UI also set new performance standards, with the MM1.5-UI model outperforming models ten times its size.

Conclusion

MM1.5 highlights that careful data curation and a staged training approach can push the boundaries of MLLM performance even at smaller scales. Future work aims to unify these specialized capabilities into a single "omni-model" to eliminate the need for task-specific fine-tuning.

Summary updated

BibTeX
@inproceedings{zhang2025mm,
  author = {Haotian Zhang and Mingfei Gao and Zhe Gan and Philipp Dufter and Nina Wenzel and Forrest Huang and Dhruti Shah and Xianzhi Du and Bowen Zhang and Yanghao Li and Sam Dodge and Keen You and Zhen Yang and Aleksei Timofeev and Mingze Xu and Hong-You Chen and Jean-Philippe Fauconnier and Zhengfeng Lai and Haoxuan You and Zirui Wang and et al.},
  title = {{MM1.5: Methods, Analysis \& Insights from Multimodal LLM Fine-tuning}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=HVtu26XDAA},
}
Figure