LMEME at SemEval-2024 Task 4: Teacher Student Fusion - Integrating CLIP with LLMs for Enhanced Persuasion Detection

Shiyi Li, Yike Wang, Liang Yang, Shaowu Zhang, Hongfei LinNAACL 2024

TL;DR

This paper introduces LMEME, a novel "Teacher-Student Fusion" system for detecting persuasion techniques in memes. By using a Large Language Model (LLM) as a teacher to generate reasoning rationales and then fine-tuning a smaller CLIP-based student model, LMEME achieved state-of-the-art results on the SemEval-2024 Task 4, outperforming 19 other systems in English.

What Problem Does This Paper Address?

Memes are a powerful medium for spreading disinformation, often employing subtle rhetorical and psychological tactics to persuade audiences. Detecting these persuasion techniques is crucial for combating the spread of false information. However, memes present a unique challenge due to their multimodal nature, where the interplay between text and image can create implicit meanings that are difficult for traditional models to grasp. Existing methods often treat text and image independently or simply concatenate their features, failing to capture the deeper, nuanced relationships that characterize persuasive memes. This paper aims to bridge this gap by developing a system that can effectively understand and reason about the persuasive intent embedded within memes.

Methods

The core of LMEME is a "Teacher-Student Fusion" framework.

  • Teacher Model (LLM for Abductive Reasoning): A Large Language Model (LLM) acts as the "teacher." It's prompted with meme text, an image caption (generated by a separate captioning model), and the ground truth label (whether persuasion is present). The LLM's task is to generate a "streamlined rationale" explaining why the meme is persuasive, drawing on its vast background knowledge of persuasion techniques without explicitly stating the label. This process is framed as abductive reasoning.
  • Student Model (CLIP-based Fusion): A smaller, more efficient student model is then trained. This model uses CLIP to encode both the meme's text and image into shared embeddings. A cross-attention mechanism is employed to align and fuse these multimodal features, allowing the model to dynamically weigh the importance of different parts of the text and image.
  • Knowledge Distillation: The rationales generated by the LLM teacher are used to guide the training of the student model. The student model is trained to minimize the cross-entropy loss between its own generated features and the teacher's rationales. This effectively "distills" the LLM's reasoning capabilities into the smaller model.
  • Label Prediction: Finally, the student model is fine-tuned for the binary classification task (persuasion vs. no persuasion) using a standard cross-entropy loss on the ground truth labels.

Datasets

The experiments were conducted on the SemEval-2024 Task 4 dataset, which includes memes potentially employing persuasion techniques. The task is multilingual, with test datasets in English, Bulgarian, North Macedonian, and Arabic. The authors specifically participated in Subtask 2b, a binary classification task.

Key Insights and Ideas

  • LLMs as Reasoning Teachers: The paper highlights the power of LLMs not just for generation, but for providing structured, explanatory rationales that can significantly improve downstream model performance. This moves beyond simple feature extraction to a more cognitive approach.
  • Bridging Modalities with Abductive Reasoning: The core innovation lies in using LLMs to bridge the gap between visual and textual information in memes through explicit reasoning. The generated rationales act as a crucial intermediate step, helping the student model understand the implicit connections.
  • Teacher-Student Fusion for Efficiency: By distilling knowledge from a large LLM into a smaller, CLIP-based model, the system achieves high performance while remaining computationally more efficient for inference.
  • Cross-Attention for Multimodal Alignment: The use of cross-attention is shown to be effective in aligning semantic features between text and visuals, leading to better persuasion detection.

Key Results

  • English Performance: LMEME achieved a Macro-F1 score of 0.8103 on Subtask 2b in English, ranking 1st out of 20 systems.
  • Multilingual Performance: The system also demonstrated strong zero-shot capabilities in other languages:
  • Bulgarian: Ranked 1/15.
  • North Macedonian: Ranked 3/15.
  • Arabic: Ranked 14/15. The authors attribute the lower performance in Arabic to its agglutinative features, which might be less effectively captured by a model primarily trained on English.
  • Ablation Study: Results showed that incorporating rationales from the LLM teacher ("LMEME") significantly improved performance over a model without rationales ("LMEME(w/o llm)") across all tested languages, validating the effectiveness of the proposed fusion approach.

Limitations and Future Work

  • Arabic Performance: The authors acknowledge that the model's performance in Arabic was not as strong as in other languages, suggesting that further research into handling agglutinative language features might be beneficial.
  • LLM Size and Prompting: The paper mentions exploring LLMs of different sizes and types, as well as different prompting strategies, as future work. This could potentially lead to even richer background knowledge generation.
  • Alternative Fusion Methods: The authors suggest exploring alternative encoding methods and better-aligned image and textual semantic fusion methods.

Summary updated

BibTeX
@inproceedings{li2024lmeme,
  author = {Shiyi Li and Yike Wang and Liang Yang and Shaowu Zhang and Hongfei Lin},
  title = {{LMEME at SemEval-2024 Task 4: Teacher Student Fusion - Integrating CLIP with LLMs for Enhanced Persuasion Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure