Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, Yueting Zhuang

TL;DR

This paper introduces VPG-C (Visual Prompt Generator Complete), a lightweight module that enables Multimodal Large Language Models (MLLMs) to "think twice" by re-attending to missing visual details needed for complex reasoning. By using a novel synthetic discriminative training strategy and the new DEMON benchmark, the authors show that VPG-C significantly improves an MLLM's ability to follow interleaved, multi-image "demonstrative instructions" without requiring expensive human-annotated data.

What Problem Does This Paper Address?

Current MLLMs are typically trained on millions of image-caption pairs. While this makes them great at identifying primary subjects, it creates an "inductive bias": the models learn to ignore fine-grained details that aren't necessary for a simple caption but are crucial for complex tasks.

The authors identify a failure in demonstrative instructions—tasks involving multiple, interleaved images and text where the context of one image is required to understand another. For example, a model might fail to determine if a product is genuine because it didn't look closely at a tiny logo detail that distinguishes it from an official reference image. Existing models lack "reasoning-aware" attention; they don't know which details to look for until they've started reasoning about the task.

Figure 1: An example of InstructBLIP (Dai et al., 2023) and our MLLM enhanced by VPG-C.

Methods

The authors propose two main innovations:

  1. VPG-C Architecture: Instead of a single pass, VPG-C uses a multi-stage approach. First, a standard Visual Prompt Generator (VPG) extracts primary features. Then, at an intermediate layer of the LLM (the "thinking" stage), the model generates instruction-specific guidance. This guidance is sent back to the VPG to extract residual visual details—the bits previously ignored. These details are then reintegrated into the LLM via a skip connection. The VPG-C architecture intercepts intermediate LLM layers to guide the extraction of missing details.
Figure 2: An overview of VPG-C.
  1. Synthetic Discriminative Training: To train VPG-C without human-labeled "detail" data, the authors developed a clever pipeline. They identify areas an existing model ignores (via attention maps), use generative tools (like Blended Diffusion) to edit those ignored areas (e.g., changing a color or removing a small object), and then train the model to describe the difference between the original and the edited image. This forces the model to learn to see what it previously missed. The synthetic training pipeline uses generative editing on "ignored" areas to create discriminative tasks.
Figure 3: Pipeline demonstration of synthetic discriminative training strategy for VPG-C.

Datasets

  • DEMON Benchmark (New): A comprehensive benchmark for demonstrative instruction understanding, featuring 31 tasks across 7 categories (e.g., Multimodal Dialogue, Visual Storytelling, Knowledge-Grounded QA). It uses interleaved image-text sequences.
  • Training Data: 500k image-caption pairs from CC3M for alignment and 64k synthetic images generated for the discriminative training task.
  • Evaluation Benchmarks: MME (for perception and cognition) and OwlEval (for human-centric evaluation of general-purpose generation).

Key Insights and Ideas

  • The "Intermediate Layer" Sweet Spot: The authors found that inserting VPG-C at the middle layers of the LLM (e.g., layer 16 of a 32-layer model) works best. Too early, and the model hasn't gathered enough context to provide guidance; too late, and there aren't enough layers left to reason with the new details. Performance peaks when VPG-C is inserted in the middle layers of the LLM.

  • Sensitivity to Image Order: Unlike many SOTA models that treat interleaved images as a "bag of frames," VPG-C is highly sensitive to the order of images. This suggests it is actually following the semantic flow of the "demonstration" rather than just looking at the images in isolation.

  • Efficiency: VPG-C is extremely parameter-efficient, adding only about 6.3M tunable parameters (0.09% of the total model). It can be trained in just 7 hours on a single A100 GPU.
Figure 6: Performance on DEMON with different insertion layers.

Key Results

  • DEMON Benchmark: VPG-C consistently outperformed advanced models like InstructBLIP and LLaVA across all categories. Notably, it surpassed SOTA by 3.92% in Multimodal Dialogue.
  • MME Benchmark: Achieved superior scores in both Perception (1299.24) and Cognition (321.07) compared to supervised instruction-tuned models.
  • Human Evaluation: In the OwlEval benchmark, VPG-C received a score of 4.19/5, significantly higher than mPLUG-Owl (3.62) and MiniGPT-4 (3.24). Human evaluators consistently ranked VPG-C higher than other open-source MLLMs.
Figure 5: Human evaluation.

Limitations and Future Work

  • Image-Choice Questions: The authors note that all tested models, including VPG-C, struggle with instructions where the options themselves are images (multimodal cloze tasks).
  • Backbone Dependency: While VPG-C is generic, its ultimate reasoning capability is still tied to the underlying frozen LLM (e.g., Vicuna or LLaMA2).
  • Future Direction: The authors hope the community will use the DEMON benchmark to improve how models handle complex, multi-step multimodal reasoning.

Summary updated

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. Our benchmark, code, and pre-trained models are available at https://github.com/DCDmllm/Cheetah.
BibTeX
@inproceedings{li2024fine,
  author = {Juncheng Li and Kaihang Pan and Zhiqi Ge and Minghe Gao and Wei Ji and Wenqiao Zhang and Tat-Seng Chua and Siliang Tang and Hanwang Zhang and Yueting Zhuang},
  title = {{Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=BXY6fe7q31},
}
Figure