SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning

Andrew C. Li, Rahul Thapa, Rahul Chalamala, Qingyang Wu, Kezhen Chen, James Zou

Abstract

Vision-Language Models (VLMs) excel at understanding single images, aided by high-quality instruction datasets. However, multi-image reasoning remains underexplored in the open-source community due to two key challenges: (1) scaling datasets with correlated images and complex reasoning instructions is resource-intensive, and (2) robust evaluation benchmarks for multi-image tasks are lacking. To address this, we introduce SMiR, a synthetic data-generation pipeline for multi-image reasoning, along with a high-quality dataset generated using this pipeline. SMiR efficiently extracts correlated images via multimodal embeddings, integrates visual and descriptive information, and leverages open-source LLMs to generate quality instructions. Using this approach, we produce 160K synthetic training samples, offering a cost-effective alternative to closed-source solutions. Additionally, we present SMiR-Bench, a multi-image reasoning benchmark comprising 200 diverse examples across seven complex reasoning tasks. SMiR-Bench is multi-turn and employs a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMiR by fine-tuning open-source VLMs and evaluating them on SMiR-Bench.

Introduction

While Vision-Language Models (VLMs) have become remarkably proficient at describing single images, they often struggle with reasoning across multiple images—such as comparing the mood of different scenes or identifying a sequence of events in a photo series. The authors identify two primary bottlenecks: existing open-source datasets often pair unrelated images or lack complex instructions, and current benchmarks rely on multiple-choice questions that fail to capture the nuance of a model’s reasoning process.

The SMIR Pipeline

The researchers developed the SMIR (Synthetic Multi-Image Reasoning) pipeline to address these data quality issues. The pipeline consists of three main stages:

  1. Multimodal Embedding Construction: SMIR creates a fused embedding by combining SigLIP/CLIP image features with their corresponding caption embeddings, ensuring that image clusters reflect both visual and semantic similarity.
  2. Random Sampling with Iteration: To maintain diversity while ensuring correlation, a probabilistic sampling algorithm selects nearby points in the embedding space.
  3. Synthetic Instruction Generation: Correlated image clusters are fed into an open-source LLM (Meta Llama 3.1 70B Instruct) to generate complex, multi-turn conversations.
The SMIR pipeline workflow, from image-caption pairs to LLM QA generation.

This approach is approximately 50x cheaper and 10x faster than using closed-source models like GPT-4. By leveraging open-source LLMs, the pipeline can generate high-quality reasoning chains that significantly improve the performance of smaller VLMs.

Example of a multi-turn reasoning dialogue generated by the SMIR pipeline.

Datasets and Evaluation

Using the SMIR pipeline, the authors generated 160,000 synthetic training samples from the ShareGPT4V dataset. To evaluate these models, they introduced SMIR-BENCH, a benchmark consisting of 200 diverse examples across seven tasks, including bird identification, pattern recognition, and storytelling. SMIR-BENCH employs a multi-turn format and uses a VLM judge to evaluate free-form responses, providing a more realistic assessment of interactive assistant performance.

Key Results and Insights

Fine-tuning open-source models on the SMIR dataset resulted in an 8% performance boost on SMIR-BENCH. Key takeaways include:

  • Correlation is Key: Multi-image reasoning is most effective when images are semantically related, forcing models to learn subtle differentiations.
  • Open-Source Scalability: High-quality synthetic data can be generated using well-prompted open-source LLMs, offering a scalable path for community development.
  • The Frontier Gap: Despite improvements, a significant gap remains between open-source models (scoring ~58.0) and closed-source leaders like GPT-4o (scoring ~98.0).

Conclusion

SMIR demonstrates a cost-effective method for scaling multi-image reasoning data. While challenges remain regarding computational overhead and absolute performance compared to proprietary models, the pipeline provides a robust foundation for future open-source VLM development.

Summary updated

Abstract

Vision-Language Models (VLMs) excel at understanding single images, aided by high-quality instruction datasets. However, multi-image reasoning remains underexplored in the open-source community due to two key challenges: (1) scaling datasets with correlated images and complex reasoning instructions is resource-intensive, and (2) robust evaluation benchmarks for multi-image tasks are lacking. To address this, we introduce SMiR, a synthetic data-generation pipeline for multi-image reasoning, along with a high-quality dataset generated using this pipeline. SMiR efficiently extracts correlated images via multimodal embeddings, integrates visual and descriptive information, and leverages open-source LLMs to generate quality instructions. Using this approach, we produce 160K synthetic training samples, offering a cost-effective alternative to closed-source solutions. Additionally, we present SMiR-Bench, a multi-image reasoning benchmark comprising 200 diverse examples across seven complex reasoning tasks. SMiR-Bench is multi-turn and employs a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMiR by fine-tuning open-source VLMs and evaluating them on SMiR-Bench.
BibTeX
@misc{li2025smir,
  author = {Andrew C. Li and Rahul Thapa and Rahul Chalamala and Qingyang Wu and Kezhen Chen and James Zou},
  title = {{SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning}},
  year = {2025},
  eprint = {2501.03675},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.03675},
}
Figure