Abstract
Recent advancements in Multimodal Large Language Models (MLLMs) utilize Visual Prompt Generators (VPGs) to bridge visual features and language tokens. However, standard training on image-caption pairs often leads to a "lazy observer" bias, where VPGs focus only on primary visual content and ignore subtle details. This paper introduces VPG-C (Visual Prompt Generator Complete), a lightweight module designed to infer and complete missing details essential for comprehending complex, interleaved "demonstrative instructions." We also propose a synthetic discriminative training strategy that eliminates the need for human-annotated data. Evaluated on our new DEMON benchmark, VPG-C demonstrates significant zero-shot performance gains and superior efficiency.
The Challenge of Demonstrative Instructions
While current MLLMs excel at simple image-to-text tasks, they often fail when faced with demonstrative instructions—tasks requiring reasoning across multiple, interleaved images and text. The root cause is the training objective: image-captioning encourages models to identify the main subject (e.g., "a milk carton") while neglecting specific attributes (e.g., a logo or quantity label) needed for comparative reasoning.
VPG-C: A "Think Twice" Architecture
To overcome the "lazy observer" effect, we propose the VPG-C module, a 6.3M parameter component that implements a two-step reasoning process:
- Initial Pass: The LLM generates a preliminary context based on coarse visual features.
- Guidance Extraction: Information is extracted from an intermediate LLM layer (optimally layer 16 for a 32-layer model) to serve as "guidance."
- Detail Completion: This guidance directs the VPG to re-examine the image for "residual visual details." These details are then integrated back into the LLM via a skip connection, allowing for a more nuanced final output.
Synthetic Discriminative Training
Training such a module typically requires expensive human-labeled data. We bypass this with a synthetic pipeline:
- Identification: We find areas typically ignored by base models.
- Editing: Using Segment Anything and Blended Diffusion, we modify these areas (e.g., swapping a logo).
- Discriminative Task: The model is trained to "spot the difference" between original and edited images, forcing it to attend to the very details it previously ignored.
Evaluation and Results
We introduced DEMON, a benchmark with 31 tasks across categories like Visual Storytelling and Multi-Image Reasoning. VPG-C achieved state-of-the-art results, outperforming larger models in multimodal dialogue by nearly 4%.
Key findings include:
- Efficiency: VPG-C tunes only 0.09% of total parameters and trains in just 7 hours on a single A100.
- Order Sensitivity: Unlike many models, VPG-C is sensitive to image order, suggesting it treats interleaved sequences as coherent narratives.
- Generalization: Improvements were also noted on standard benchmarks like MME and OwlEval.
Conclusion
VPG-C provides a scalable, efficient solution for enhancing the detail-oriented reasoning of MLLMs. By enabling models to "think twice," we bridge the gap between simple captioning and complex, context-aware instruction following.
Summary updated