TL;DR
This paper introduces MAPPER, a novel framework for understanding figurative language in multimodal contexts (images and text). MAPPER uses a "describer" to generate detailed image descriptions and a "thinker" to leverage these descriptions, the original claim, and the image for prediction and explanation, achieving state-of-the-art results.
What Problem Does This Paper Address?
Figurative language, especially in media like memes, art, and comics, presents a significant challenge for AI. Accurately determining whether a textual claim complements or contradicts an accompanying image, and then explaining why, requires deep understanding of not just visual and textual content, but also social common sense and cultural contexts. Traditional methods often struggle with this abstract reasoning, leading to performance gaps compared to humans. This paper aims to bridge this gap by enhancing the model's ability to interpret and explain multimodal figurative language.
Methods
MAPPER is a two-component framework:
- Describer: This component uses a frozen large vision-language model (LLM) to generate a detailed textual description of the image. This description acts as a "modal supplement," providing richer semantic information than the image alone might offer to the subsequent stage. The LLM's weights are kept frozen, meaning it's used for its pre-trained descriptive capabilities.
- Thinker: This component is a fine-tuned large multi-modal model. It takes the image description from the describer, the original textual claim, and the image itself as input. It then uses a specially designed prompt template to predict whether the claim entails or contradicts the image and generates a textual explanation for its prediction. The thinker is trained to unify classification and explanation generation into a single task.
The core novelty lies in the explicit "modal supplement" provided by the describer, which aims to enrich the input for the thinker, and the structured prompt-based approach for the thinker to perform both prediction and explanation.
Datasets
The experiments were conducted on the V-FLUTE dataset. This dataset is described as a compilation of five smaller datasets from previous work on visual metaphor and multimodal understanding, supplemented with annotated explanations. The paper notes that it followed dataset splits from the "UNDERSTANDING OF FIGURATIVE LANGUAGE THROUGH VISUAL ENTAILMENT" competition.
Key Insights and Ideas
- Textual Supplementation is Key: The paper strongly suggests that providing a detailed textual description of an image, generated by a separate model, significantly aids in understanding figurative language. This "modal supplement" helps the main model to better grasp the nuances of the image.
- Prompt Engineering Matters: The design of the prompt template for the "thinker" component was found to be crucial. Removing this prompt template (PTR2) led to a drastic drop in performance, highlighting its importance in guiding the model's reasoning and output generation.
- Fine-tuning Boosts Performance: Even with a relatively small dataset and a simple fine-tuning approach (like LoRA), the model's performance on figurative language understanding can be significantly improved, narrowing the gap with human capabilities.
- Zero-shot Gemini vs. Text-only: The observation that Gemini-Pro-Vision performed better than Gemini (Text only) in zero-shot conditions suggests that while direct image input is beneficial, replacing images with their descriptions might lead to information loss in certain prediction tasks.
Key Results
MAPPER achieved state-of-the-art performance on the V-FLUTE dataset.
- Overall Evaluation: The paper claims MAPPER ranked top 1 in overall evaluation, exceeding the second-place by 28.57%.
- Specific Metrics: On the V-FLUTE dataset, MAPPER achieved F1@0 of 89.67, F1@50 of 89.09, and F1@60 of 74.15.
- Comparison to Zero-Shot: Compared to prompt-based zero-shot methods, MAPPER improved performance by 28.57% (F1@0), 34.33% (F1@50), and 50% (F1@60) after supervised fine-tuning.
- Qualitative Example: In a case study, MAPPER's explanation for "The cat is busy as a bee" was more aligned with the ground truth and had a higher BERT-score than baseline methods, demonstrating its ability to generate accurate and relevant explanations.
Limitations and Future Work
- Describer's Impact: The describer component had a relatively small impact on F1@0 and F1@50 scores, and even a slight increase in F1@60. The authors acknowledge this might indicate limitations in the describer for complex cases.
- Prompt Importance: The PTR2 prompt template was found to be critical, with its removal causing a significant performance drop. The paper notes they did not explore different prompts to test robustness.
- Model Scale: The authors suggest further exploration into whether MLLMs with a larger number of parameters could achieve more accurate judgment and understanding of figurative language in images.
- Competition Constraints: Due to time constraints, they did not fully explore why the textual modal supplement might negatively impact F1@60.
Summary updated