Empowering Vision-Language Models to Follow Interleaved Vision-Language Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Yueting Zhuang

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) utilize Visual Prompt Generators (VPGs) to bridge visual features and language tokens. However, standard training on image-caption pairs often leads to a "lazy observer" bias, where VPGs focus only on primary visual content and ignore subtle details. This paper introduces VPG-C (Visual Prompt Generator Complete), a lightweight module designed to infer and complete missing details essential for comprehending complex, interleaved "demonstrative instructions." We also propose a synthetic discriminative training strategy that eliminates the need for human-annotated data. Evaluated on our new DEMON benchmark, VPG-C demonstrates significant zero-shot performance gains and superior efficiency.

The Challenge of Demonstrative Instructions

While current MLLMs excel at simple image-to-text tasks, they often fail when faced with demonstrative instructions—tasks requiring reasoning across multiple, interleaved images and text. The root cause is the training objective: image-captioning encourages models to identify the main subject (e.g., "a milk carton") while neglecting specific attributes (e.g., a logo or quantity label) needed for comparative reasoning.

Visual attention on specific details like brand and quantity labels, which are often missed by standard MLLMs.

VPG-C: A "Think Twice" Architecture

To overcome the "lazy observer" effect, we propose the VPG-C module, a 6.3M parameter component that implements a two-step reasoning process:

  1. Initial Pass: The LLM generates a preliminary context based on coarse visual features.
  2. Guidance Extraction: Information is extracted from an intermediate LLM layer (optimally layer 16 for a 32-layer model) to serve as "guidance."
  3. Detail Completion: This guidance directs the VPG to re-examine the image for "residual visual details." These details are then integrated back into the LLM via a skip connection, allowing for a more nuanced final output.

Synthetic Discriminative Training

Training such a module typically requires expensive human-labeled data. We bypass this with a synthetic pipeline:

  • Identification: We find areas typically ignored by base models.
  • Editing: Using Segment Anything and Blended Diffusion, we modify these areas (e.g., swapping a logo).
  • Discriminative Task: The model is trained to "spot the difference" between original and edited images, forcing it to attend to the very details it previously ignored.

Evaluation and Results

We introduced DEMON, a benchmark with 31 tasks across categories like Visual Storytelling and Multi-Image Reasoning. VPG-C achieved state-of-the-art results, outperforming larger models in multimodal dialogue by nearly 4%.

The concept of Demonstrative Instruction involves interleaved multimodal context.

Key findings include:

  • Efficiency: VPG-C tunes only 0.09% of total parameters and trains in just 7 hours on a single A100.
  • Order Sensitivity: Unlike many models, VPG-C is sensitive to image order, suggesting it treats interleaved sequences as coherent narratives.
  • Generalization: Improvements were also noted on standard benchmarks like MME and OwlEval.

Conclusion

VPG-C provides a scalable, efficient solution for enhancing the detail-oriented reasoning of MLLMs. By enabling models to "think twice," we bridge the gap between simple captioning and complex, context-aware instruction following.

Summary updated

Abstract

Recent advancements in Multimodal Large Language Models (MLLMs) have been utilizing Visual Prompt Generators (VPGs) to convert visual features into tokens that LLMs can recognize. This is achieved by training the VPGs on millions of image-caption pairs, where the VPG-generated tokens of images are fed into a frozen LLM to generate the corresponding captions. However, this image-captioning based training objective inherently biases the VPG to concentrate solely on the primary visual contents sufficient for caption generation, often neglecting other visual details. This shortcoming results in MLLMs' underperformance in comprehending demonstrative instructions consisting of multiple, interleaved, and multimodal instructions that demonstrate the required context to complete a task. To address this issue, we introduce a generic and lightweight Visual Prompt Generator Complete module (VPG-C), which can infer and complete the missing details essential for comprehending demonstrative instructions. Further, we propose a synthetic discriminative training strategy to fine-tune VPG-C, eliminating the need for supervised demonstrative instructions. As for evaluation, we build DEMON, a comprehensive benchmark for demonstrative instruction understanding. Synthetically trained with the proposed strategy, VPG-C achieves significantly stronger zero-shot performance across all tasks of DEMON. Further evaluation on the MME and OwlEval benchmarks also demonstrate the superiority of VPG-C. Our benchmark, code, and pre-trained models are available at https://github.com/DCDmllm/Cheetah.
BibTeX
@inproceedings{li2024empowering,
  author = {Juncheng Li and Kaihang Pan and Zhiqi Ge and Minghe Gao and Hanwang Zhang and Wei Ji and Wenqiao Zhang and Tat-Seng Chua and Siliang Tang and Yueting Zhuang},
  title = {{Empowering Vision-Language Models to Follow Interleaved Vision-Language Instructions}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=BXY6fe7q31},
}
Figure