Abstract
While modern Vision-Language (VL) models demonstrate impressive performance on high-level reasoning benchmarks like Visual Commonsense Reasoning (VCR), their underlying understanding remains questionable. This paper identifies a "prediction inconsistency" where models succeed at complex inferences but fail at the basic visual or factual components supporting those inferences. To address this, we propose Multimodal Evaluation (ME), an automated framework for generating fine-grained sub-questions, and Multimodal Coaching (MC), a training methodology that leverages these sub-questions to enhance model robustness and consistency.
Introduction
Visual Commonsense Reasoning (VCR) is a challenging task requiring models to not only identify objects but also explain the rationale behind their interactions. Despite high leaderboard scores, models often rely on linguistic shortcuts or shallow correlations. For instance, a model might correctly infer a "kidnapping" event while failing to recognize the simple act of "grabbing." This discrepancy highlights a lack of integrated multimodal understanding, where models make "lucky guesses" without grasping the underlying scene primitives.
Methodology
The authors developed a two-part framework to diagnose and improve model reasoning: Multimodal Evaluation (ME) and Multimodal Coaching (MC).
Multimodal Evaluation (ME) Pipeline
The ME pipeline automates the generation of "check-your-work" sub-questions. It begins by converting images and VCR questions into Visual, Textual, and Background Knowledge statements. These are parsed into Subject-Verb-Object (SVO) triplets to form a unified Multimodal Graph. From this graph, the system generates multiple-choice questions using templates and sophisticated distractor generation (via GPT-2 and T5), filtered for quality and relevance.
Multimodal Coaching (MC)
MC integrates these sub-questions into the training process. By identifying and reinforcing the specific modalities (vision, text, or knowledge) where a model fails, MC forces the model to develop a more consistent and grounded understanding of the scene. If a model fails a simple sub-question, that question is added to the training pool to reinforce that specific modality.
Datasets and Benchmark
The study introduces the Multimodal Sub-question Evaluation Benchmark, derived from 110,000 VCR images. It contains 630,000 multiple-choice sub-questions categorized into Visual (110k), Textual (260k), and Background Knowledge (260k) types. Human verification via Amazon Mechanical Turk confirmed high agreement between AI-generated and human-verified questions, ensuring the benchmark's reliability.
Key Insights and Results
- The Consistency Gap: Models frequently suffer from the "Lucky Guess" phenomenon. Consistency between high-level reasoning and low-level facts can drop by up to 40% when models are forced to answer both correctly.
- Directionality of Learning: A major finding is that learning low-level visual primitives (e.g., "holding a bag") facilitates high-level reasoning (e.g., "shopping"), but the reverse is not observed. High-level reasoning doesn't automatically grant low-level understanding.
- Performance and Coaching Success: Top models like VL-BERT and UNITER saw an accuracy drop of about 20% on the ME benchmark compared to standard VCR performance. However, applying the MC framework improved VL-BERT’s VCR accuracy by over 1% and its sub-question performance by a massive 7%, while significantly increasing self-consistency.
Limitations and Future Work
The ME pipeline currently relies on image captions, which can introduce cascading errors if automated generators are used. Additionally, the system is optimized for English SVO structures. Future research should focus on reducing caption dependency and better bridging the gap between external knowledge graphs (like ConceptNet) and specific visual contexts to achieve more human-like reasoning.
Summary updated