Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning

Bardia Safaei, Faizan Siddiqui, Jiacong Xu, Vishal M. Patel, Shao-Yuan Lo

Abstract

Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets.

Introduction

Visual Instruction Tuning (VIT) is the cornerstone of modern Large Vision-Language Models (LVLMs) like LLaVA, enabling them to follow complex human instructions regarding visual content. However, creating these datasets is prohibitively expensive. Generating high-quality text instructions for millions of images requires either intensive human labor or costly calls to proprietary APIs like GPT-4.

Current data selection research focuses on reducing training time by identifying a "smart" subset of existing data. However, these methods suffer from a fundamental "chicken-and-egg" problem: they require the instructions to be already present to evaluate the quality of a sample. Consequently, researchers must still pay the full cost of labeling an entire dataset before they can prune it.

Comparison of PreSel vs. existing data selection paradigms.

To resolve this, we propose PreSel (Pre-Instruction Data Selection). PreSel flips the traditional pipeline by selecting the most impactful unlabeled images before any instructions are generated. This paradigm shift reduces instruction-generation costs by 85% while maintaining nearly 100% of the performance of full-scale training.

Methodology

The PreSel framework identifies the most beneficial images in a large unlabeled pool through a two-stage process:

1. Task-Importance Estimation

The model begins with a small "reference set" (5% of the total data) for which instructions are generated. It then calculates an Instruction Relevance Score (IRS). The IRS quantifies how much a text question aids the model in predicting the correct response compared to seeing the image in isolation. If a model's performance significantly improves with the instruction, that task is deemed "important" and receives a higher sampling budget.

2. Task-wise Cluster-based Selection

With budgets established for various tasks (e.g., OCR, VQA, Captioning), PreSel employs a lightweight vision encoder (DINOv2) to extract features from the remaining unlabeled images. These images are clustered, and a Neighbor Centrality (NC) score is used to select the most representative images from each cluster. This ensures the final 15% subset is visually diverse and representative of the entire dataset distribution.

The PreSel framework: Task-Importance Estimation and Cluster-based Selection.

Experimental Results

We evaluated PreSel on two major VIT datasets: LLaVA-1.5 (over 600,000 samples) and Vision-Flan (191 distinct tasks). Performance was measured across standard benchmarks including VQAv2, ScienceQA, MME, and POPE.

  • Efficiency and Performance: Using only 15% of the data, PreSel achieved 97.9% of the performance of the full LLaVA-1.5 dataset. On the diverse Vision-Flan dataset, PreSel actually surpassed the full-data baseline, reaching 100.1% relative performance.
  • Cost Savings: PreSel's selection process is highly efficient, requiring only 9 GPU-hours on LLaVA-1.5, compared to 55.5 hours for COINCIDE and 73.5 hours for Self-Filter.
  • Instruction Necessity: The IRS metric revealed that more data is not always better; for certain tasks, adding redundant data can actually degrade performance. Smart selection based on representativeness often yields superior generalization compared to "hard" example mining.

Conclusion

PreSel demonstrates that visual diversity is a powerful proxy for instructional value. By decoupling selection from labeling, it provides a practical path for resource-constrained researchers to build high-performing LVLMs. Future work will explore estimating task importance without any initial instructions, potentially using zero-shot classifiers or metadata to further reduce the "seed" cost.

Project Page: https://bardisafa.github.io/PreSel

Summary updated

Abstract

Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets. The link to our project page: https://bardisafa.github.io/PreSel
BibTeX
@inproceedings{safaei2025filter,
  author = {Bardia Safaei and Faizan Siddiqui and Jiacong Xu and Vishal M. Patel and Shao-Yuan Lo},
  title = {{Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=nEAqL32Kzk},
}
Figure