Abstract
Visual instruction tuning (VIT) for large vision-language models (LVLMs) requires training on expansive datasets of image-instruction pairs, which can be costly. Recent efforts in VIT data selection aim to select a small subset of high-quality image-instruction pairs, reducing VIT runtime while maintaining performance comparable to full-scale training. However, a major challenge often overlooked is that generating instructions from unlabeled images for VIT is highly expensive. We introduce PreSel (Pre-Instruction Data Selection), a more practical paradigm that selects the most beneficial unlabeled images and generates instructions only for the selected images. PreSel reduces instruction generation costs by 85% while achieving 98% of full-data performance on the LLaVA-1.5 benchmark.
The Bottleneck of Visual Instruction Tuning
Visual Instruction Tuning is the process of fine-tuning LVLMs to follow human commands (e.g., "Describe the colors of the bus in this image"). To do this well, researchers typically need hundreds of thousands of image-instruction pairs.
The primary bottleneck is not the images, but the instructions. Generating high-quality instructions requires either expensive human labor or costly API calls to models like GPT-4. While recent data selection methods attempt to find the best subset of data to train on, they almost all assume that instructions for the entire dataset already exist. This creates a "chicken-and-egg" problem: researchers must pay to annotate everything just to figure out which 10-15% they actually needed.
The PreSel Framework
PreSel addresses this inefficiency by operating primarily on unlabeled images through a two-stage framework:
- Task-Importance Estimation: The model randomly samples a tiny "reference set" (5%) of the data and generates instructions for them. It then calculates an Instruction Relevance Score (IRS), which measures how much the linguistic instruction actually helps the model predict the answer compared to seeing the image alone. Tasks where the instruction is vital receive a higher sampling budget.
- Task-wise Cluster-based Selection: For the remaining 95% of unlabeled images, PreSel uses a lightweight vision encoder (DINOv2) to extract features. Images within each task are clustered using k-means, and the model selects the most "representative" images using a Neighbor Centrality (NC) score—picking images that are most similar to their neighbors in feature space rather than outliers.
Key Results and Efficiency
PreSel was evaluated on major benchmarks including LLaVA-1.5 and Vision-Flan. The results demonstrate that high performance does not require massive annotation budgets:
- Performance: Using only 15% of the data, PreSel achieved an average relative performance of 97.9% on LLaVA-1.5 and slightly outperformed the full-data model (100.1%) on the high-diversity Vision-Flan dataset.
- Cost Savings: By selecting images first, PreSel reduced instruction generation costs by 85%.
- Compute Efficiency: Compared to other selection methods like COINCIDE (55.5 GPU-hr) or Self-Filter (73.5 GPU-hr), PreSel's selection process took only 9 GPU-hours.
Conclusion
PreSel proves that "more is not always better" in visual instruction tuning. By focusing on representative images and task-specific importance before annotation, researchers can drastically reduce the financial and computational barriers to training powerful Large Vision-Language Models. This approach achieves "Pareto efficiency," saving both money and time without sacrificing model capability.
Summary updated