A Report on the FigLang 2024 Shared Task on Multimodal Figurative Language

Shreyas Kulkarni, Arkadiy Saakyan, Tuhin Chakrabarty, Smaranda Muresan2024 · DOI

Abstract

We present the outcomes of the Multimodal Figurative Language Shared Task held at the 4th Workshop on Figurative Language Processing (FigLang 2024) co-located at NAACL 2024. The task utilized the V-FLUTE dataset (Saakyan et al., 2024) which is comprised of <image, text> pairs that use figurative language and includes detailed textual explanations for the entailment or contradiction relationship of each pair. The challenge for participants was to develop models capable of accurately identifying the visual entailment relationship in these multimodal instances and generating persuasive free-text explanations. The results showed that the participants' models significantly outperformed the initial baselines in both automated and human evaluations. All participating systems outperformed the LLaVA-ZS baseline, provided by us in F1-score.

Introduction

While Large Multimodal Models (LVMs) have become remarkably good at literal tasks—like identifying objects or reading text—they often stumble when language turns figurative. A literal model might struggle to interpret an image where a mundane event is depicted through a complex visual metaphor. For instance, if a faculty meeting is shown as a medieval battlefield, a model must understand the implied comparison to correctly evaluate a claim like "the meeting was peaceful."

A visual metaphor depicting a faculty meeting as a medieval battlefield.

The FigLang 2024 Shared Task addresses this gap in explainable figurative visual entailment. This involves two core challenges:

  1. Classification: Determining if an image (the premise) entails or contradicts a figurative statement (the hypothesis).
  2. Explanation: Generating a human-readable, persuasive reason for that decision, moving beyond simple classification toward verifiable reasoning.

Methodology and Evaluation

The shared task invited teams to build models capable of handling complex multimodal inputs. Evaluation utilized the F1@X metric to ensure models weren't just "guessing" correctly:

  • F1@0 measures basic label accuracy.
  • F1@50 and F1@60 only count a prediction as correct if the generated explanation achieves a high BERTScore (semantic similarity) compared to a human gold-standard explanation.

Two notable architectural approaches emerged from the competition. The winning system, MAPPER, utilized a "describer-thinker" framework where a frozen LLaVA model first extracts literal semantic details, which are then processed by a fine-tuned "thinker" model. Another approach, FigCLIP, used a bidirectional fusion module to align figurative meanings between CLIP and GPT-2.

The V-FLUTE Dataset

The competition centered on V-FLUTE (Visual Figurative Language Understanding with Textual Explanations), a dataset of 5,115 multimodal pairs. It is a "mega-dataset" compiled from five specialized sources, covering a wide range of figurative types.

Overview of the V-FLUTE dataset subsets and figurative language types.

As shown in the table above, the dataset includes diverse subsets such as IRFL (metaphors and idioms in captions), VisMet (visual metaphors), MemeCap (humor and memes), MuSE (sarcasm), and NYCC (complex visual puns from the New Yorker Caption Contest).

Results and Analysis

The competition results showed a clear hierarchy in performance, highlighting the difficulty of the task for general-purpose models. MAPPER took the top spot with an F1@0 of 0.90 and an F1@60 of 0.75, showing robust reasoning and explanation capabilities. In contrast, GPT-4V achieved an F1@0 of 0.70, falling behind specialized models, while the zero-shot baseline (LLaVA-ZS) trailed significantly at 0.45.

Key insights include:

  • The "Describer-Thinker" Advantage: Success suggests that breaking the problem into literal feature extraction followed by figurative reasoning is more effective than end-to-end processing.
  • Zero-Shot Limitations: The poor performance of general-purpose models indicates that figurative understanding is a specific skill requiring targeted fine-tuning.
  • Explanation as a Bottleneck: There is a noticeable drop-off between simple accuracy (F1@0) and high-quality explanation (F1@60) across all models, highlighting that "persuasive reasoning" remains a significant hurdle for LVMs.

Conclusion

The FigLang 2024 Shared Task demonstrated that while figurative reasoning is a challenge for general-purpose LVMs, specialized architectures and targeted datasets like V-FLUTE can significantly bridge the gap. Future work will likely focus on improving the persuasiveness of free-text explanations and exploring whether these capabilities can be distilled into smaller, more efficient models.

Summary updated

Abstract

We present the outcomes of the Multimodal Figurative Language Shared Task held at the 4th Workshop on Figurative Language Processing (FigLang 2024) co-located at NAACL 2024.The task utilized the V-FLUTE dataset (Saakyan et al., 2024) which is comprised of <image, text> pairs that use figurative language and includes detailed textual explanations for the entailment or contradiction relationship of each pair.The challenge for participants was to develop models capable of accurately identifying the visual entailment relationship in these multimodal instances and generating persuasive free-text explanations.The results showed that the participants' models significantly outperformed the initial baselines in both automated and human evaluations.We also provide an overview of the systems submitted and analyze the results of the evaluations.All participating systems outperformed the LLaVA-ZS baseline, provided by us in F1-score.
BibTeX
@misc{kulkarni2024report,
  author = {Shreyas Kulkarni and Arkadiy Saakyan and Tuhin Chakrabarty and Smaranda Muresan},
  title = {{A Report on the FigLang 2024 Shared Task on Multimodal Figurative Language}},
  year = {2024},
  doi = {10.18653/v1/2024.figlang-1.16},
}
Figure