Multi-Modal Large Language Model Enhanced Pseudo 3D Perception Framework for Visual Commonsense Reasoning

Jian Zhu, Hanli Wang, Miaojing Shi

TL;DR

This paper introduces MEP3P, a novel framework that enhances visual commonsense reasoning (VCR) by incorporating pseudo 3D perception and leveraging multi-modal large language models (MLLMs). By considering object depth and 3D relationships, MEP3P significantly improves the model's ability to understand visual scenes and answer complex questions, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Visual Commonsense Reasoning (VCR) is a challenging task that requires models to not only identify objects in an image but also understand their relationships and infer the underlying context to answer questions and provide justifications. Existing VCR approaches often rely on 2D object detection and struggle to accurately capture the spatial relationships between objects, especially when depth and 3D information are crucial for human-like understanding. For instance, two objects might appear close in a 2D image but be far apart in 3D space, leading to misinterpretations. Furthermore, while Multi-Modal Large Language Models (MLLMs) have shown great promise in vision-language tasks, they are typically trained on entire scenes and not specifically adapted for the fine-grained, object-centric reasoning required by VCR. This paper aims to bridge this gap by developing a framework that integrates pseudo 3D perception with MLLM capabilities for more robust VCR.

Methods

The proposed framework, MEP3P, consists of two main components: a Pseudo 3D Perception VCR framework (P3PV) and an MLLM Enhanced Reasoning module (MER).

  • P3PV: This component enhances visual features by incorporating pseudo 3D information.
  • Depth-enhanced Visual Features: It uses monocular depth estimation to infer object depths and then combines these with 2D bounding box information to create pseudo 3D positions. These features, along with original visual features, are fed into a depth-aware Transformer.
  • Depth-aware Transformer: This novel Transformer architecture modifies the attention mechanism to consider depth differences between objects. This allows the model to better associate objects that are semantically related and have similar depths, or to distinguish between objects that might appear close in 2D but are far apart in 3D. It also incorporates depth-aware attention from objects to words in the answer, assigning pseudo depths to answer words based on related objects.
  • MER: This module leverages MLLMs to improve cross-modal alignment and reasoning.
  • MLLM Integration: The BLIP-2 model is used to extract aligned cross-modal features. To handle referring expressions in VCR tasks (e.g., "[person 1]"), these are replaced with generic semantic labels (e.g., "person") before being fed into the MLLM.
  • Parameter Optimization: A parameter optimization technique is introduced to evaluate the quality of data batches based on multi-level reasoning confidence, allowing for more effective model training.

Datasets

The experiments were conducted on the VCR benchmark dataset. This dataset is derived from movie scenes and contains approximately 290k four-way multi-choice QA problems, with answers and rationales that are mixtures of visual and linguistic elements.

Key Insights and Ideas

  • Depth is Crucial for Object Relations: The paper strongly emphasizes that object depth and 3D spatial relationships are critical for accurate visual commonsense reasoning, going beyond simple 2D proximity.
  • Depth-Aware Attention: The novel depth-aware Transformer effectively captures nuanced object relationships by considering depth differences, leading to better scene understanding.
  • MLLMs for VCR: While MLLMs are powerful, they require adaptation for VCR's specific needs, particularly regarding referring expressions and fine-grained object reasoning. The MER module demonstrates how to effectively integrate MLLMs for this task.
  • Pseudo 3D Perception: The concept of "pseudo 3D" perception, derived from monocular depth estimation, provides a practical way to inject 3D understanding without requiring full 3D scene reconstruction.

Key Results

The proposed MEP3P framework achieved state-of-the-art performance on the VCR benchmark dataset.

  • Overall Performance: MEP3P achieved an accuracy of 77.6% for Q → A, 77.0% for QA → R, and 60.0% for Q → AR.
  • Ablation Studies:
    • The depth-enhanced visual features (VFE) and depth-aware Transformer (DT) contributed significantly, improving performance by up to 1.0% for the Q → AR task compared to a baseline with only 2D positions.
  • The MLLM Enhanced Reasoning (MER) module provided a substantial boost, with the full MEP3P model (Base+VFE+DT+POMRC+MER) outperforming other baselines by an average of around 9%.
  • Efficiency: The MEP3P method requires significantly less computational resources for training (2 Tesla V100 GPUs for ~30 hours) compared to other vision-and-language models.

Limitations and Future Work

  • Limitations:
    • The paper notes that the MLLM component, when used alone, performs poorly in VCR because it struggles to distinguish specific visual objects referred to in text.
  • The parameter optimization technique relies on confidence thresholds, which might require careful tuning.
  • Future Work:
    • The authors suggest applying the depth-aware Transformer to other 3D tasks, such as autonomous driving scene understanding.
  • They propose exploring more effective techniques to integrate task-specific knowledge into MLLMs.
  • A future direction is to develop a unified framework for various vision-and-language tasks using the MLLM-enhanced pseudo 3D perception approach.

Summary updated

Abstract

The visual commonsense reasoning (VCR) task is to choose an answer and provide a justifying rationale based on the given image and textural question. Representative works first recognize objects in images and then associate them with key words in texts. However, existing approaches do not consider exact positions of objects in a human-like three-dimensional (3D) manner, making them incompetent to accurately distinguish objects and understand visual relation. Recently, multi-modal large language models (MLLMs) have been used as powerful tools for several multi-modal tasks but not for VCR yet, which requires elaborate reasoning on specific visual objects referred by texts. In light of the above, an MLLM enhanced pseudo 3D perception framework is designed for VCR. Specifically, we first demonstrate that the relation between objects is relevant to object depths in images, and hence introduce object depth into VCR frameworks to infer 3D positions of objects in images. Then, a depth-aware Transformer is proposed to encode depth differences between objects into the attention mechanism of Transformer to discriminatively associate objects with visual scenes guided by depth. To further associate the answer with the depth of visual scene, each word in the answer is tagged with a pseudo depth to realize depth-aware association between answer words and objects. On the other hand, BLIP-2 as an MLLM is employed to process images and texts, and the referring expressions in texts involving specific visual objects are modified with linguistic object labels to serve as comprehensible MLLM inputs. Finally, a parameter optimization technique is devised to fully consider the quality of data batches based on multi-level reasoning confidence. Experiments on the VCR dataset demonstrate the superiority of the proposed framework over state-of-the-art approaches.
BibTeX
@article{zhu2024multi,
  author = {Jian Zhu and Hanli Wang and Miaojing Shi},
  title = {{Multi-Modal Large Language Model Enhanced Pseudo 3D Perception Framework for Visual Commonsense Reasoning}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  doi = {10.1109/tcsvt.2024.3428487},
}
Figure