InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4

Lai Wei, Zihao Jiang, Weiran Huang, Lichao SunSubmission to ICLR 2024 · Rejected

TL;DR

Borrowing the "less is more" philosophy from text-only LLMs (like LIMA), InstructionGPT-4 demonstrates that fine-tuning a multimodal model on just 200 meticulously selected instruction-following examples—roughly 6% of MiniGPT-4’s original alignment set—consistently outperforms the full-dataset baseline. The authors achieve this by training a small, learnable data selector that automatically scores and curates high-quality vision-language data across diverse semantic clusters.

What Problem Does This Paper Address?

Multimodal Large Language Models (MLLMs) such as MiniGPT-4 and LLaVA rely on a two-stage training paradigm: large-scale pre-training on image-text pairs followed by supervised fine-tuning on instruction data. However, common instruction-tuning sets are riddled with irrelevant, incomplete, or hallucinated responses that degrade downstream conversational and reasoning performance.

While manual curation (as done in LIMA) works well, human annotation is expensive and difficult to scale across multimodal tasks. Standard data pruning heuristics (like classification loss or sample difficulty) also fail to translate effectively to generative multimodal tasks. The authors address how to automatically identify and curate high-quality multimodal instruction data to achieve stronger alignment with minimal data.

Methods

The authors propose a principled, learning-based data curation pipeline combining quality assessment and diversity preservation:

  1. Multimodal Quality Indicators: Every image-instruction-response triplet is mapped to a feature embedding combining:
  2. CLIP Score: Measures semantic consistency between the image and response.
  3. Length Score: Penalizes excessively verbose or overly terse responses.
  4. Reward Score: Evaluated via an OpenAssistant reward model to gauge human preference.
  5. GPT Score: GPT-4-evaluated response rating assessing grammar, fluency, and reasoning.
  6. Multimodal Features: Low-dimensional PCA-reduced representations derived from frozen ViT and LLaMA-2 encoders.
  7. Ground-Truth "Genuine Quality Labels": The authors partition the candidate pool into 30 subsets using K-means++, fine-tune an MLLM on each subset, and evaluate each checkpoint across four VQA validation sets. The resulting average accuracy is treated as the cluster's ground-truth alignment quality label.
  8. Trainable Data Selector: A 2-layer self-attention network fits the concatenated indicator embeddings to these empirical quality labels.
  9. Diversity-Aware Selection: Before final scoring, the original dataset is grouped into $K = 10$ semantic clusters via spectral clustering on image embeddings. The trained selector ranks samples within each cluster, extracting the top candidates proportionally to assemble a final diverse, high-quality set of 200 samples.
Candidate Pool (3.4k) ──> Spectral Clustering ──> Score with Self-Attention Selector ──> Top 200 Triplets ──> Fine-Tune MiniGPT-4

Datasets

  • Training & Selection Source: The cc_sbu_align dataset (3,439 image-instruction pairs used in MiniGPT-4 Stage 2). Scalability was also validated on the detail_10k dataset.
  • Genuine Quality Labeling Sets: GQA, IconQA, OKVQA, and ScienceQA.
  • Downstream Evaluation Benchmarks:
  • MME Benchmark: Perception and cognition across 14 subtasks.
  • MMBench: 20 ability dimensions across 6 reasoning levels.
  • VQA Benchmarks (LVLM-eHub): DocVQA, TextVQA, STVQA, and VizWiz.
  • LLaVA-Bench: 60 complex open-ended visual questions judged by GPT-4.

Key Insights and Ideas

  • Quality Dominates Quantity in Multimodal Alignment: Scaling down the alignment dataset from 3,439 to 200 samples improved performance across almost all evaluated metrics, proving that alignment is primarily a formatting/style-learning phase rather than a knowledge-acquisition phase.
  • The Pitfall of Larger Subsets: In ablation studies, as curated sample size grew from 200 up to 1,000, the performance advantage over random sampling narrowed. Adding more samples inevitably reintroduces lower-quality data that harms alignment.
  • Heuristics Alone Are Insufficient: Combining multiple complementary indicators (reward scores, CLIP similarity, length, and dense multimodal embeddings) through a self-attention model significantly outperformed any single indicator or classical pruning technique (such as EL2N or prototypicality clustering).
  • Zero-Shot Transferability: Even though the selector was trained on VQA-derived quality signals, the curated 200 samples boosted performance on out-of-domain benchmarks like MME and MMBench, and transferred successfully to other architectures like Qwen-VL without retraining.

Key Results

  • MME Benchmark: InstructionGPT-4 scored 648.26, outperforming original MiniGPT-4 (625.20) and a 200-sample random baseline (527.26). It beat MiniGPT-4 in 8 out of 14 subtasks (with a striking jump from 23.50 to 50.50 in Artwork perception).
  • MMBench: Achieved 31.42 vs. MiniGPT-4’s 29.87 (and random selection’s 23.95), improving on 13 of 20 capability dimensions.
  • VQA Tasks: Averaged 22.30% zero-shot accuracy across STVQA, VizWiz, DocVQA, and TextVQA, beating MiniGPT-4 (20.54%).
  • GPT-4 Pairwise Evaluation (LLaVA-Bench): Across 60 questions evaluated in both prompt orders to remove position bias, InstructionGPT-4 won 26, tied 18, and lost only 16 matchups against MiniGPT-4.

Limitations and Future Work

  • Initial Computational Overhead: Generating "genuine quality labels" requires training multiple model checkpoints on data partitions, which can be computationally demanding to set up initially.
  • Indicator Space: The selector relies on a pre-defined set of hand-picked metrics (CLIP, GPT-4 scoring, Reward models). The authors note that exploring other signals—such as perplexity or error L2-norm—remains open for future study.
  • Dataset Diversity: Experiments centered primarily on MiniGPT-4's alignment data and LLaVA's detail_10k. Extending automated instruction mining across larger, messier open-web multimodal sets is an identified future direction.

Summary updated

Abstract

Multimodal large language models are typically trained in two stages: first pre-training on image-text pairs, and then fine-tuning using supervised vision-language instruction data. Recent studies have shown that large language models can achieve satisfactory results even with a limited amount of high-quality instruction-following data. In this paper, we introduce InstructionGPT-4, which is fine-tuned on a small dataset comprising only 200 examples, amounting to approximately 6% of the instruction-following data used in the alignment dataset for MiniGPT-4. To achieve this, we first propose several metrics to access the quality of multimodal instruction data. Based on these metrics, we present an effective and trainable data selector to automatically identify and filter low-quality vision-language data. By employing this method, InstructionGPT-4 outperforms the original MiniGPT-4 on various evaluations. Overall, our findings demonstrate that less but high-quality instruction tuning data is efficient in enabling multimodal large language models to generate better output.
BibTeX
@misc{wei2023instructiongpt,
  author = {Lai Wei and Zihao Jiang and Weiran Huang and Lichao Sun},
  title = {{InstructionGPT-4: A 200-Instruction Paradigm for Fine-Tuning MiniGPT-4}},
  year = {2023},
  eprint = {2308.12067},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12067},
}
Figure