Volta at SemEval-2021 Task 6: Towards Detecting Persuasive Texts and Images using Textual and Multimodal Ensemble

Kshitij Gupta, Devansh Gautam, Radhika MamidiACL 2021

TL;DR

This paper tackles the challenge of detecting persuasive techniques in memes, a multimodal task that combines text and images. The authors propose a transfer learning approach using BERT-based models and explore ensembling textual and visual-linguistic models, achieving competitive results on the SemEval-2021 Task 6.

What Problem Does This Paper Address?

Memes have become a powerful medium for spreading information and influencing public opinion, often employing subtle rhetorical and psychological tactics. Detecting these persuasive techniques, especially in a multimodal format like memes, is crucial for understanding online discourse and combating disinformation. SemEval-2021 Task 6 specifically addresses this by focusing on the detection of persuasion techniques in memes, offering three subtasks: textual multi-label classification, textual multi-label classification with span identification, and multimodal classification using both text and images. The challenge lies in effectively integrating information from both modalities and handling the inherent complexity and potential subtlety of persuasive language and imagery.

Methods

The authors leverage transfer learning with pre-trained transformer models, specifically BERT and RoBERTa, for the textual tasks (Task A and B). For Task A (textual multi-label classification), they fine-tune BERT/RoBERTa models by attaching a feed-forward network to the [CLS] token embedding. Task B, which includes span identification, is also modeled as a token-level multi-label classification problem, adapting the textual models to predict spans.

For the multimodal Task C, they experiment with Visual-Linguistic (VL) models such as UNITER, VisualBERT, and LXMERT. Initially, they find that these VL models, when trained directly on multimodal data, underperform purely textual models. This leads them to investigate an ensembling approach. Their proposed ensemble architecture concatenates the [CLS] token embeddings from independently trained textual and VL models, followed by a classification layer. This allows each modality's model to focus on its strengths before combining their representations.

Datasets

The paper uses the dataset provided for SemEval-2021 Task 6. This dataset consists of 951 memes, divided into training, development, and test splits. The dataset is annotated with labels for persuasive techniques. The paper notes that while the memes are the same across tasks, the labeling differs: Task A uses only textual cues, Task B adds span identification for textual cues, and Task C incorporates visual cues for a broader set of techniques.

Key Insights and Ideas

  • Textual cues are surprisingly powerful: The authors discover that purely textual models often perform competitively, and sometimes even better, than multimodal models on tasks involving memes. This suggests that many persuasive techniques are conveyed primarily through text, even when an image is present.
  • VL models struggle with meme-specific nuances: Despite their potential, off-the-shelf VL models don't always excel at meme classification, possibly due to the unique nature of meme humor, sarcasm, and cultural references that might not be captured by general-purpose VL training.
  • Ensembling offers a robust solution: Combining the strengths of specialized textual models with VL models through ensembling proves to be an effective strategy, leading to improved performance by leveraging the best of both worlds.
  • Class imbalance is a significant challenge: The dataset exhibits class imbalance, necessitating techniques like weighted loss functions to ensure that less frequent persuasive techniques are not overlooked.

Key Results

The paper reports the following F1-micro scores on the test set:

  • Task A (Textual Classification): 57.0
  • Task B (Textual Classification + Span ID): 48.2
  • Task C (Multimodal Classification): 52.1

Specifically, RoBERTa LARGE is highlighted as the best-performing model for Task A and B. For Task C, their ensembled model (RoBERTa LARGE + UNITER) achieved the reported score. The authors also note that their submitted model for Task C performed worse on the test set due to using a different checkpoint than the best validation checkpoint.

Limitations and Future Work

The authors acknowledge that their ensembling method is basic and suggest that exploring more sophisticated architectures could further improve performance. They also point out that while many persuasive techniques can be identified using text alone, future work could focus on improving the detection of techniques that are truly multimodal in nature. The VL models are still considered to be in their "nascent stages," implying room for significant advancement in this area.

Summary updated

BibTeX
@inproceedings{gupta2021volta,
  author = {Kshitij Gupta and Devansh Gautam and Radhika Mamidi},
  title = {{Volta at SemEval-2021 Task 6: Towards Detecting Persuasive Texts and Images using Textual and Multimodal Ensemble}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure