Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning

Bardia Safaei, Faizan Siddiqui, Jiacong Xu, Vishal M. Patel, Shao-Yuan Lo

Abstract

Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. We introduce PreSel (Pre-Instruction Data Selection), a more practical paradigm that selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel reduces instruction generation costs by 85% while achieving 98% of full-data performance on the LLaVA-1.5 benchmark.

The Bottleneck of Visual Instruction Tuning

Visual Instruction Tuning is the process of fine-tuning LVLMs to follow human commands (e.g., "Describe the colors of the bus in this image"). To do this well, researchers typically need hundreds of thousands of image-instruction pairs.

The primary bottleneck is not the images, but the instructions. Generating high-quality instructions requires either expensive human labor or costly API calls to models like GPT-4. While recent data selection methods attempt to find the best subset of data to train on, they almost all assume that instructions for the entire dataset already exist. This creates a "chicken-and-egg" problem: researchers must pay to annotate everything just to figure out which 10-15% they actually needed.

PreSel shifts the data selection process to occur before instruction generation, significantly reducing costs.

The PreSel Framework

PreSel addresses this inefficiency by operating primarily on unlabeled images through a two-stage framework:

  1. Task-Importance Estimation: The model randomly samples a tiny "reference set" (5%) of the data and generates instructions for them. It then calculates an Instruction Relevance Score (IRS), which measures how much the linguistic instruction actually helps the model predict the answer compared to seeing the image alone. Tasks where the instruction is vital receive a higher sampling budget.
  2. Task-wise Cluster-based Selection: For the remaining 95% of unlabeled images, PreSel uses a lightweight vision encoder (DINOv2) to extract features. Images within each task are clustered using k-means, and the model selects the most "representative" images using a Neighbor Centrality (NC) score—picking images that are most similar to their neighbors in feature space rather than outliers.
Overview of the PreSel framework, detailing the Task-Importance Estimation and visual feature clustering processes.

Key Results and Efficiency

PreSel was evaluated on major benchmarks including LLaVA-1.5 and Vision-Flan. The results demonstrate that high performance does not require massive annotation budgets:

  • Performance: Using only 15% of the data, PreSel achieved an average relative performance of 97.9% on LLaVA-1.5 and slightly outperformed the full-data model (100.1%) on the high-diversity Vision-Flan dataset.
  • Cost Savings: By selecting images first, PreSel reduced instruction generation costs by 85%.
  • Compute Efficiency: Compared to other selection methods like COINCIDE (55.5 GPU-hr) or Self-Filter (73.5 GPU-hr), PreSel's selection process took only 9 GPU-hours.

Conclusion

PreSel proves that "more is not always better" in visual instruction tuning. By focusing on representative images and task-specific importance before annotation, researchers can drastically reduce the financial and computational barriers to training powerful Large Vision-Language Models. This approach achieves "Pareto efficiency," saving both money and time without sacrificing model capability.

Summary updated

Abstract

Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. Most existing VIT datasets rely heavily on human annotations or paid services like the GPT API, which limits users with constrained resources from creating VIT datasets for custom applications. To address this, we introduce Pre-Instruction Data Selection (PreSel), a more practical data selection paradigm that directly selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel first estimates the relative importance of each vision task within VIT datasets to derive task-wise sampling budgets. It then clusters image features within each task, selecting the most representative images with the budget. This approach reduces computational overhead for both instruction generation during VIT data formation and LVLM fine-tuning. By generating instructions for only 15% of the images, PreSel achieves performance comparable to full-data VIT on the LLaVA-1.5 and Vision-Flan datasets. The link to our project page: https://bardisafa.github.io/PreSel
BibTeX
@inproceedings{safaei2025filter,
  author = {Bardia Safaei and Faizan Siddiqui and Jiacong Xu and Vishal M. Patel and Shao-Yuan Lo},
  title = {{Filter Images First, Generate Instructions Later: Pre-Instruction Data Selection for Visual Instruction Tuning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=nEAqL32Kzk},
}
Figure