DIEM: Decomposition-Integration Enhancing Multimodal Insights

Xinyi Jiang, Guoming Wang, Junhao Guo, Juncheng Li, Wenqiao Zhang, Rongxing Lu, Siliang TangCVPR 2024 · DOI

Introduction

Multimodal Large Language Models (MLLMs), such as LLaVA and GPT-4V, have demonstrated impressive capabilities in processing both text and imagery. However, these models often struggle with complex reasoning tasks that require fine-grained attention to specific visual details. A common failure mode is the presence of "visual noise"—redundant or irrelevant information in a global image that distracts the model from the core problem. While text-based Chain-of-Thought (CoT) methods have improved logical reasoning, they typically treat the image as a static, global input, failing to achieve precise multimodal alignment.

To address this, we introduce DIEM (Decomposition-Integration Enhancing Multimodal Insights), a training-free, "plug-and-play" framework. DIEM mimics human cognitive processes by decomposing complex multimodal problems into simpler sub-tasks and localized visual regions, then integrating these insights for a final, accurate answer.

Methodology

The DIEM framework operates through a three-stage pipeline designed to enhance fine-grained alignment between textual reasoning and visual evidence.

Detailed walkthrough of the DIEM pipeline, illustrating the decomposition of images and questions into matched sub-components.

1. Text-Visual Decomposition

The process begins by breaking down the input into manageable components:

  • Textual Decomposition: An LLM (e.g., GPT-3.5-Turbo) decomposes the original complex question into a sequence of simpler, logical sub-questions.
  • Visual Decomposition: The original image is partitioned into multiple sub-images or segments using the Segment Anything Model (SAM), identifying distinct visual entities.

2. Text-Visual Matching

For each generated sub-question, the system identifies the most relevant visual context. Using a CLIP model, similarity scores are calculated between the sub-question and all visual segments. The top-$k$ most relevant sub-images are selected to provide the necessary visual evidence for that specific reasoning step.

3. Answer Generation and Integration

The MLLM (e.g., LLaVA-v1.5) processes each sub-question sequentially. It generates a "sub-answer" by considering the selected sub-images alongside the original global image for context. Finally, all sub-question/sub-answer pairs are concatenated into a comprehensive prompt. The model then integrates this structured reasoning chain to produce the final answer to the original question.

Experimental Evaluation

DIEM was evaluated on two rigorous benchmarks: ScienceQA, which focuses on multi-subject science questions, and MM-Vet, which tests 16 emergent multimodal capabilities including spatial awareness and OCR.

Key Results

DIEM consistently outperformed the standard LLaVA-v1.5 baseline across both datasets:

  • ScienceQA: The framework achieved a +2.03% average accuracy boost. Notably, in tasks involving image-based context, accuracy improved by 3.40% (from 65.70% to 69.10%).
  • MM-Vet: The total score rose from 31.1 to 32.4. The most significant improvements were observed in Spatial Awareness (+4.9) and Recognition (+2.2), validating the effectiveness of the sub-image approach in helping models focus on critical visual details.

Analysis and Insights

The Impact of Sub-Questions and Sub-Images

Ablation studies reveal a strong synergy between textual and visual decomposition. Decomposing both the question (SQ) and the image (SI) yields better results than decomposing either in isolation. This suggests that localized vision and localized text are highly complementary in complex reasoning.

Performance analysis showing the impact of different k values and the synergy between sub-questions (SQ) and sub-images.

The "Goldilocks Zone" of $k$

The number of sub-images ($k$) provided to the model is a critical hyperparameter. As shown in the performance trends, providing too few sub-images ($k=1$) limits necessary context, while providing all segments ($k=ALL$) reintroduces visual noise. For the ScienceQA dataset, $k=5$ was identified as the optimal balance for maximizing accuracy.

Conclusion and Future Work

DIEM provides a robust, training-free method to enhance MLLM performance by structuring multimodal inputs to mirror human-like problem-solving. While effective, the framework's performance is currently dependent on the quality of external decomposition models like SAM and GPT-3.5. Future work will focus on improving image-text matching techniques and exploring the use of more advanced LLMs to further refine the reasoning chain. Additionally, addressing the computational overhead of sequential sub-question answering remains a priority for real-time applications.

Summary updated

BibTeX
@inproceedings{jiang2024diem,
  author = {Xinyi Jiang and Guoming Wang and Junhao Guo and Juncheng Li and Wenqiao Zhang and Rongxing Lu and Siliang Tang},
  title = {{DIEM: Decomposition-Integration Enhancing Multimodal Insights}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02578},
}
Figure