Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense

Zhecan Wang, Haoxuan You, Yicheng He, Wenhao Li, Kai-Wei Chang, Shih-Fu Chang

Abstract

While modern Vision-Language (VL) models demonstrate impressive performance on high-level reasoning benchmarks like Visual Commonsense Reasoning (VCR), their underlying understanding remains questionable. This paper identifies a "prediction inconsistency" where models succeed at complex inferences but fail at the basic visual or factual components supporting those inferences. To address this, we propose Multimodal Evaluation (ME), an automated framework for generating fine-grained sub-questions, and Multimodal Coaching (MC), a training methodology that leverages these sub-questions to enhance model robustness and consistency.

Introduction

Visual Commonsense Reasoning (VCR) is a challenging task requiring models to not only identify objects but also explain the rationale behind their interactions. Despite high leaderboard scores, models often rely on linguistic shortcuts or shallow correlations. For instance, a model might correctly infer a "kidnapping" event while failing to recognize the simple act of "grabbing." This discrepancy highlights a lack of integrated multimodal understanding, where models make "lucky guesses" without grasping the underlying scene primitives.

Example of prediction inconsistency where a model correctly identifies a high-level action (Q1) but fails on basic visual (Q2) and knowledge-based (Q3, Q4) sub-questions.

Methodology

The authors developed a two-part framework to diagnose and improve model reasoning: Multimodal Evaluation (ME) and Multimodal Coaching (MC).

Multimodal Evaluation (ME) Pipeline

The ME pipeline automates the generation of "check-your-work" sub-questions. It begins by converting images and VCR questions into Visual, Textual, and Background Knowledge statements. These are parsed into Subject-Verb-Object (SVO) triplets to form a unified Multimodal Graph. From this graph, the system generates multiple-choice questions using templates and sophisticated distractor generation (via GPT-2 and T5), filtered for quality and relevance.

The Multimodal Evaluation (ME) pipeline: parsing multimodal inputs into a unified graph to generate fine-grained sub-questions across visual, textual, and knowledge domains.

Multimodal Coaching (MC)

MC integrates these sub-questions into the training process. By identifying and reinforcing the specific modalities (vision, text, or knowledge) where a model fails, MC forces the model to develop a more consistent and grounded understanding of the scene. If a model fails a simple sub-question, that question is added to the training pool to reinforce that specific modality.

Datasets and Benchmark

The study introduces the Multimodal Sub-question Evaluation Benchmark, derived from 110,000 VCR images. It contains 630,000 multiple-choice sub-questions categorized into Visual (110k), Textual (260k), and Background Knowledge (260k) types. Human verification via Amazon Mechanical Turk confirmed high agreement between AI-generated and human-verified questions, ensuring the benchmark's reliability.

Key Insights and Results

  • The Consistency Gap: Models frequently suffer from the "Lucky Guess" phenomenon. Consistency between high-level reasoning and low-level facts can drop by up to 40% when models are forced to answer both correctly.
  • Directionality of Learning: A major finding is that learning low-level visual primitives (e.g., "holding a bag") facilitates high-level reasoning (e.g., "shopping"), but the reverse is not observed. High-level reasoning doesn't automatically grant low-level understanding.
  • Performance and Coaching Success: Top models like VL-BERT and UNITER saw an accuracy drop of about 20% on the ME benchmark compared to standard VCR performance. However, applying the MC framework improved VL-BERT’s VCR accuracy by over 1% and its sub-question performance by a massive 7%, while significantly increasing self-consistency.

Limitations and Future Work

The ME pipeline currently relies on image captions, which can introduce cascading errors if automated generators are used. Additionally, the system is optimized for English SVO structures. Future research should focus on reducing caption dependency and better bridging the gap between external knowledge graphs (like ConceptNet) and specific visual contexts to achieve more human-like reasoning.

Summary updated

Abstract

Visual commonsense understanding requires Vision Language (VL) models to not only understand image and text but also cross-reference in-between to fully integrate and achieve comprehension of the visual scene described. Recently, various approaches have been developed and have achieved high performance on visual commonsense benchmarks. However, it is unclear whether the models really understand the visual scene and underlying commonsense knowledge due to limited evaluation data resources. To provide an in-depth analysis, we present a Multimodal Evaluation (ME) pipeline to automatically generate question-answer pairs to test models' understanding of the visual scene, text, and related knowledge. We then take a step further to show that training with the ME data boosts the model's performance in standard VCR evaluation. Lastly, our in-depth analysis and comparison reveal interesting findings: (1) semantically low-level information can assist the learning of high-level information but not the opposite; (2) visual information is generally under utilization compared with text.
BibTeX
@inproceedings{wang2022understanding,
  author = {Zhecan Wang and Haoxuan You and Yicheng He and Wenhao Li and Kai-Wei Chang and Shih-Fu Chang},
  title = {{Understanding ME? Multimodal Evaluation for Fine-grained Visual Commonsense}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.626},
}
Figure