DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models

Biao Wu, Yiwu Zhong, Meng Fang, Ling Chen

Here is a concise, engaging summary of the paper "DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models".

TL;DR

Training large vision-language models (VLMs) on massive instruction datasets is computationally expensive and often plagued by noisy or redundant data. DOSE introduces a zero-shot data selection framework that leverages off-the-shelf, pretrained models to evaluate text quality and image-text alignment without any task-specific training. By combining these scores into a joint distribution and using a diversity-aware weighted sampling strategy, DOSE matches or exceeds full-dataset performance using only 20% of the data.

What Problem Does This Paper Address?

Visual Instruction Tuning (VIT) is the critical second stage of training Large Vision-Language Models (LVLMs), enabling them to follow human prompts across multimodal tasks. However, training on massive instruction datasets is incredibly resource-intensive.

To cut costs, researchers use data selection methods to filter out low-quality samples. Existing approaches fall into two flawed categories:

  1. Simple Proxies: Methods relying on basic signals like loss, perplexity, or gradient norms. These are efficient but only capture local training dynamics, missing broader semantic richness.
  2. Seen-Data Selectors: Methods that use full-fledged VLMs as quality evaluators. While accurate, they require expensive pre-training or fine-tuning on the target data itself, adding massive computational overhead and risking data contamination.

DOSE addresses this dilemma: Can we identify the most valuable data for visual instruction tuning using off-the-shelf models that have never seen the target data, completely bypassing task-specific training?

Methods

DOSE splits the data selection process into two main phases: Quality Assessment and Weighted Random Sampling.

[Input Data] ---> [Vicuna-7B]   ---> Text Quality Score \
            ---> [CLIP-ViT]    ---> Image-Text Score   ---> [Joint 2D Distribution] ---> [Weighted Random Sampling] ---> [20% Core Subset]

1. Off-the-Shelf Quality Assessment

Instead of training a custom evaluator, DOSE uses two readily available models:

  • Text Quality Score: An instruction-tuned LLM (Vicuna-7B) is prompted in a zero-shot manner to judge if a text sample is well-formatted, informative, and safe. The softmax probability of the model answering "yes" serves as the quality score.
  • Image-Text Relevance Score: The widely adopted CLIP model (CLIP-ViT-B/32) computes the cosine similarity between image and text embeddings to ensure the visual and textual components actually align.

2. Weighted Random Sampling (WRS)

A common pitfall in data filtering is rigid "Top-$K$" truncation, which discards rare but highly informative long-tail samples. DOSE avoids this by constructing a joint 2D distribution of the text and image scores.

Using Kernel Density Estimation (KDE), the authors shift the target sampling probability toward high-quality regions while maintaining a non-zero probability across the entire score spectrum. This ensures the final subset prioritizes high-quality data without sacrificing semantic diversity.

Figure 1: Comparison of data selection methods. (A) Methods based on single-score proxies, often derived from either language or vision signals. (B) Methods that use VLMs as quality evaluators, which may suffer from data contamination or prior exposure when the evaluator has been trained on overlapping corpora. (C) Our method combines off-the-shelf pretrained models without requiring task-specific training on the target dataset.

Datasets

The authors validated DOSE by filtering two primary training sets:

  • LLaVA Stage 2 Dataset: 665K multimodal instruction-following triples.
  • MathV360k: A specialized dataset for multimodal mathematical reasoning.

The filtered subsets were evaluated across nine diverse benchmarks:

  • General VQA & Reasoning: VQAv2, GQA, VizWiz, SQA-I, TextVQA, POPE.
  • MMLU-style Multimodal Benchmarks: MME, MMBench (English/Chinese), LLaVA-Bench (LLaVA-W).
  • Specialized Math: MathVista.

Key Insights and Ideas

  • The Top-$K$ Trap: Purely selecting the highest-scoring data points causes models to underperform because high-scoring samples tend to cluster in semantically repetitive regions. Retaining "moderate" or "low" score data via a soft probability distribution is essential for maintaining model robustness.
  • LLMs as Neutral Referees: Using an LLM purely via forward inference provides a more global, semantically grounded perspective on text quality than raw training metrics like loss or perplexity.
  • The Power of the Upper-Right Quadrant: Ablation studies mapping the joint distribution showed that performance spikes dramatically when both CLIP scores and text quality scores are high. Filtering based on only one modality introduces significant bias.
Figure 2: Left: Combined distribution of Text-Quality Score and CLIP Score, with Text-Quality Score on the x-axis and CLIP Score on the y-axis. Color intensity indicates data density, where brighter colors correspond to higher densities. Right: Comparison between the original data distribution and the distribution after applying WRS on 665K samples from LLaVA Stage 2. The same axes as the left figure are used, with an additional z-axis representing data density. The height along the z-axis corresponds to the density in each region. The blue distribution corresponds to the target distribution q(x), while the green distribution corresponds to the original distribution p(x).

Key Results

  • 96% Performance with 20% Data: When training LLaVA-1.5-7B on just a 20% subset selected by DOSE, the model retained 96% of the performance of a model trained on 100% of the data across general benchmarks.
  • Surpassing the Full Dataset in Math: On the specialized MathVista benchmark, a model trained on a DOSE-selected 20% subset of MathV360k achieved 104.8% relative performance, actually outperforming the model trained on the full dataset by filtering out distracting noise.
  • Massive Efficiency Gains: Unlike methods like COINCIDE or ICONS, which require an expensive full-data fine-tuning pass before selection, DOSE operates entirely at inference time. It achieves a superior wall-clock time-to-performance trade-off, scaling linearly with dataset size.

Limitations and Future Work

While highly effective, the authors note that their study was constrained by a limited exploration budget. They primarily validated DOSE on the LLaVA-1.5 architecture (7B and 13B variants) and tested a fixed set of sampling ratios (mostly 20%, 40%, and 60%).

Future work remains to validate how well DOSE generalizes to more diverse, next-generation VLM architectures and whether more fine-grained sampling ratios can unlock even higher efficiency.

Summary updated

Abstract

High-quality and diverse multimodal data are essential for improving vision-language models (VLMs), yet existing datasets often contain noisy, redundant, and poorly aligned samples. To address these problems, data filtering is commonly used to enhance the efficiency and performance of multimodal learning, but it introduces extra computational cost because filtering models are usually trained on the same data they are meant to screen. To reduce this cost, we study DOSE, which explores whether off-the-shelf pretrained models that have never seen the target data can be used to select training samples for larger and stronger multimodal models without any task-specific training. Even without fine-tuning, these models can effectively assess text quality and image-text alignment to guide data selection. Based on this, we build a joint quality-alignment distribution and apply adaptive weighted sampling to select informative samples while maintaining long-tail diversity. This approach enhances data diversity, enabling models trained on DOSE-filtered data to match or surpass those trained on the full dataset on standard VQA and math benchmarks. Extensive experiments demonstrate its effectiveness, efficiency, and scalability.
BibTeX
@inproceedings{wu2026dose,
  author = {Biao Wu and Yiwu Zhong and Meng Fang and Ling Chen},
  title = {{DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.45},
}

Loading related content...

Figure