Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat‐Seng Chua, Siliang Tang, Yueting Zhuang

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C.

Introduction

Most current MLLMs, such as BLIP-2 or InstructBLIP, rely on a Visual Prompt Generator (VPG) to translate images into tokens compatible with a Large Language Model (LLM). While effective for general image description, these models often suffer from "primary subject bias." Because they are trained on broad image-caption pairs, the VPG learns to prioritize obvious subjects (e.g., "a dog") while ignoring subtle but critical details (e.g., a specific logo).

This bias is particularly problematic for demonstrative instructions—complex tasks involving multiple interleaved images and text that require fine-grained comparisons or reasoning across a sequence. For instance, identifying a counterfeit product by comparing it to a genuine reference image requires a level of visual precision that standard VPGs often lack.

The VPG-C Architecture

To overcome these limitations, we introduce VPG-C (Visual Prompt Generator Complete), a lightweight module designed to enable a "think twice" reasoning process. Instead of a single feed-forward pass, VPG-C allows the model to re-examine the visual input based on preliminary linguistic context.

The VPG-C architecture enables multi-stage reasoning by feeding intermediate LLM guidance back into the visual prompt generator to extract residual details.

The architecture functions in four stages:

  1. Initial Pass: A standard, frozen VPG extracts primary visual features.
  2. Guidance Generation: The LLM processes the text and initial visual tokens up to an intermediate layer (e.g., the midpoint of the model). It then generates a "guidance" vector based on its initial task understanding.
  3. Residual Extraction: This guidance is fed back into the VPG-C module, which re-attends to the image to extract "missing" details relevant to the specific instruction.
  4. Reintegration: These new "residual" visual tokens are injected back into the LLM via a skip connection, providing the necessary context for a final, accurate response.

Synthetic Discriminative Training

A major challenge in training for demonstrative instructions is the lack of large-scale, high-quality human-labeled data. We address this through a novel Synthetic Discriminative Training strategy.

We first identify image regions typically ignored by standard VPGs using attention maps. We then use "Segment Anything" and "Blended Diffusion" to perform local edits—adding, removing, or swapping objects—in these ignored areas. VPG-C is then trained to describe the subtle differences between the original and edited images. This forces the module to learn how to recover the fine-grained details that the base model overlooks, without requiring manual instruction data.

Evaluation and Results

We evaluated VPG-C using DEMON, a new benchmark for demonstrative instruction understanding comprising 31 tasks across 7 categories, including Multimodal Dialogue and Knowledge-Grounded QA.

Comparison of InstructBLIP and VPG-C on a demonstrative instruction task. VPG-C correctly identifies the counterfeit product by attending to the specific icon detail missed by the base model.

Key Findings

  • Superior Reasoning: VPG-C (built on Vicuna-7B) consistently outperformed much larger models. On tasks requiring visual relation inference, VPG-C scored 25.90, more than doubling InstructBLIP’s 11.49.
  • Efficiency: VPG-C adds only 6.3M tunable parameters (approx. 0.09% of the total model) and can be trained in just 7 hours on a single A100 GPU.
  • Sequence Sensitivity: Unlike many state-of-the-art models that treat interleaved images as an unordered "bag of features," VPG-C is highly sensitive to image order, proving it truly follows the logical flow of demonstrations.
  • Generalization: VPG-C also set new high bars on the MME benchmark (Perception: 1299.24, Cognition: 321.07) and was preferred by human raters in OwlEval for complex multimodal queries.

Conclusion

VPG-C demonstrates that MLLMs do not necessarily need more parameters or massive human-labeled datasets to improve reasoning. By implementing a feedback loop that allows the model to "think twice" about visual details, we can significantly enhance performance on complex, zero-shot demonstrative tasks.

Summary updated

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. Our benchmark, code, and pre-trained models are available at https://github.com/DCDmllm/Cheetah.
BibTeX
@inproceedings{li2024fine,
  author = {Juncheng Li and Kaihang Pan and Zhiqi Ge and Minghe Gao and Hanwang Zhang and Wei Ji and Wenqiao Zhang and Tat‐Seng Chua and Siliang Tang and Yueting Zhuang},
  title = {{Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=BXY6fe7q31},
}
Figure