FigCLIP: A Generative Multimodal Model with Bidirectional Cross-attention for Understanding Figurative Language via Visual Entailment

Qihao Yang, Xuelin WangNAACL 2024

TL;DR

This paper introduces FigCLIP, a novel end-to-end multimodal model that bridges the gap between visual and textual understanding for figurative language. By employing a bidirectional cross-attention mechanism and leveraging CLIP and GPT-2, FigCLIP not only predicts whether an image entails a figurative claim but also generates explanations, outperforming existing models like GPT-4V.

What Problem Does This Paper Address?

Figurative language, which includes metaphors, similes, and sarcasm, is a nuanced aspect of human communication that often relies on cultural context, abstract implications, and imagination. While traditional Natural Language Processing (NLP) models have made strides in understanding textual figurative language, incorporating visual information presents a significant challenge. The recent FigLang-2024 Multimodal Figurative Language Shared Task aims to address this by evaluating models on their ability to understand figurative language through visual entailment – determining if an image supports or contradicts a figurative claim. Existing multimodal models often struggle with this task due to the inherent complexity of figurative language and the need for deep cultural and contextual understanding. Furthermore, many current approaches treat label prediction and explanation generation as separate tasks, hindering a holistic understanding.

Methods

The proposed FigCLIP model is an end-to-end system built upon the foundations of CLIP and GPT-2. Its core innovation lies in a bidirectional fusion module with cross-attention. This module allows for a deeper interaction between visual and textual representations. Specifically, it generates a "claim-guided multimodal vector" by using the claim as the query and the image as the key/value, and vice-versa for an "image-guided multimodal vector." These vectors are then concatenated and fed into a binary classifier for label prediction (entailment or contradiction). For explanation generation, a projector maps the fused multimodal representations into GPT-2's vocabulary space, enabling it to generate coherent explanations. This integrated approach allows FigCLIP to jointly perform both prediction and generation, a significant advantage over methods that handle these tasks separately.

Datasets

The training data for FigCLIP is compiled from five datasets:

  • A subset of the Visual Metaphors dataset (731 instances).
  • A subset of textual metaphors with images (1,322 instances).
  • A subset of memes with annotated claims and explanations (853 instances).
  • A subset of sarcastic captions with images (1,000 instances).
  • A subset of unique images with captions and textual explanations (520 instances).

The evaluation is performed on the V-FLUTE test set, which is associated with the FigLang-2024 shared task.

Key Insights and Ideas

  • Bidirectional Cross-Attention is Key: The paper highlights that a simple concatenation of image and text features is insufficient. The bidirectional cross-attention mechanism is crucial for enabling a nuanced understanding of how visual elements support or contradict figurative language.
  • End-to-End is Better: By jointly training for both prediction and explanation generation, FigCLIP achieves a more cohesive understanding and avoids the limitations of decoupled approaches.
  • Resolution Matters: The study demonstrates that using higher resolution images (336x336) significantly improves performance compared to lower resolutions (224x224), suggesting that fine-grained visual details are vital for interpreting figurative language.
  • Lightweight Design: Despite its multimodal capabilities, FigCLIP is designed to be relatively lightweight, partly due to freezing the image encoder and using a carefully designed projector for mapping between CLIP and GPT-2 spaces.

Key Results

FigCLIP, initialized with CLIP-ViT-L/14@336px, achieved competitive results on the V-FLUTE test set, ranking second overall. The model attained:

  • 70% F1-score
  • 67% F1@50-score
  • 50% F1@60-score

These results are particularly impressive when compared to baselines like LlaVA (zero-shot), which achieved significantly lower scores (45% F1, 38% F1@50, 21% F1@60). Furthermore, FigCLIP outperformed GPT-4V (zero-shot) by a small margin in F1@50 and F1@60 scores, while being orders of magnitude more efficient to train.

Limitations and Future Work

The authors acknowledge that freezing the image encoder, while reducing training costs, might limit the model's ability to learn richer multimodal figurative language representations. Future work includes optimizing different layers of the image encoder and exploring larger generative language models like Llama and Vicuna to enhance FigCLIP's generalization capabilities. They also plan to investigate the impact of different generative models on performance and further optimize the trade-off between performance and computational cost.

Summary updated

BibTeX
@inproceedings{yang2024figclip,
  author = {Qihao Yang and Xuelin Wang},
  title = {{FigCLIP: A Generative Multimodal Model with Bidirectional Cross-attention for Understanding Figurative Language via Visual Entailment}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure