GreyBox at SemEval-2024 Task 4: Progressive Fine-tuning (for Multilingual Detection of Propaganda Techniques)

Nathan Roll, Calbert GrahamNAACL 2024

TL;DR

This paper introduces "GreyBox," a novel progressive fine-tuning approach for large language models (LLMs) to detect propaganda techniques in internet memes across multiple languages. By iteratively exposing LLMs to increasingly specific data, GreyBox achieves competitive performance, demonstrating the effectiveness of this method for low-resource languages and specialized tasks.

What Problem Does This Paper Address?

The digital age has seen a surge in propaganda and disinformation, particularly within internet memes. Detecting these persuasive techniques is challenging due to their often subtle, context-dependent nature and the multilingual landscape of online content. Existing methods, while improving, often struggle with the nuances of meme-based propaganda and the complexities of low-resource languages. SemEval-2024 Task 4 specifically aims to address this by focusing on the "Multilingual Detection of Persuasion Techniques in Memes," requiring systems to identify propaganda techniques from meme text and/or images. This paper tackles the challenge of adapting powerful, pre-trained LLMs to this specific, multilingual task, especially when training data for certain languages or subtasks might be limited.

Methods

The core of the GreyBox system is a progressive fine-tuning strategy applied to a pre-trained LLM (specifically, GPT 3.5-Turbo). This approach involves two distinct fine-tuning steps:

  1. Priming for Meaning: The LLM is first exposed to all available training and validation data across all subtasks. The goal here is to imbue the model with a general understanding of the context, intention, and implied meanings within memes, leveraging the broad pre-training of the LLM. This step uses a large, diverse dataset to help the model grasp the semantic nuances.
  2. Structural Adaptation: Following the initial priming, the model undergoes a second, more targeted fine-tuning phase. This step focuses exclusively on Subtask 1 data (text-only meme analysis) to align the LLM with the specific structural requirements of the output format and the precise identification of techniques.

This multi-stage approach is designed to mitigate the "catastrophic forgetting" often seen in single-stage fine-tuning, where models lose general capabilities when adapting to a new task. The authors also note that this strategy is supported by prior research showing benefits for low-resource settings and imbalanced datasets.

Datasets

The primary dataset used is from the SemEval-2024 Task 4, which consists of memes manually labeled with one of 22 propaganda techniques. The authors utilized a total of 18,650 training examples derived from 11,111 unique memes, split across training, development, and validation sets. This data was primarily in English, with the task requiring multilingual capabilities. For evaluating multilingual performance, the authors created 16 translated versions of the held-out data using the Google Translate API.

Key Insights and Ideas

  • Progressive Fine-tuning Bridges the Gap: The most significant insight is that a staged fine-tuning process, moving from broad semantic understanding to specific structural adaptation, is highly effective for adapting LLMs to specialized, multilingual tasks like meme propaganda detection.
  • English-Centric Data Still Yields Multilingual Gains: Despite fine-tuning primarily on English data, the system demonstrated performance improvements across a wide range of languages, highlighting the inherent multilingual capabilities of large pre-trained models and the effectiveness of the progressive approach in unlocking them.
  • Structural Adaptation is Crucial: The second fine-tuning stage (structural adaptation) proved to be more impactful than the initial meaning priming for many languages, suggesting that aligning the model with the specific output format and task requirements is key to maximizing performance.
  • LLMs Show Bias Towards Common Languages: Zero-shot evaluations of popular LLMs revealed a consistent bias towards more common languages or those closely related to them, underscoring the challenge for low-resource languages.

Key Results

The GreyBox system achieved competitive rankings on Subtask 1 across English, Bulgarian, and North Macedonian. The paper presents detailed zero-shot performance comparisons of GPT 3.5 Turbo, Llama 2 70B Chat, and Mixtral 8x7B Instruct across various languages, showing that while LLMs have some multilingual capabilities, performance varies significantly.

After the first fine-tuning step (meaning priming), the model showed within-language performance improvements across nearly all settings, even with English-only data. The final model, after structural adaptation, demonstrated further gains in hierarchical F1, precision, and recall for most languages, outperforming zero-shot and intermediate counterparts. English, in particular, showed the highest overall increase in performance, with Stage 2 of fine-tuning contributing the most significant gains.

Limitations and Future Work

The authors acknowledge that their fine-tuning was primarily English-centric, which might limit the extent to which the model truly "understands" the nuances of other languages. They also note that the translation process for multilingual evaluation might introduce its own artifacts.

Future work could explore:

  • The impact of hyperparameters on downstream performance.
  • A deeper understanding of how data integration processes affect performance, particularly in comparing progressive fine-tuning to single-stage methods.
  • Investigating and mitigating biases in LLMs that affect performance across diverse language communities and use cases.
  • Further research into the interplay between pre-training corpora, post-training regimes, and evaluation data.

Summary updated

BibTeX
@inproceedings{roll2024greybox,
  author = {Nathan Roll and Calbert Graham},
  title = {{GreyBox at SemEval-2024 Task 4: Progressive Fine-tuning (for Multilingual Detection of Propaganda Techniques)}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure