FPAI at SemEval-2021 Task 6: BERT-MRC for Propaganda Techniques Detection

Xiaolong Hou, Junsong Ren, Gang Rao, Lianxin Lian, Zhihao Ruan, Yang Mo, JIanping ShenACL 2021

TL;DR

This paper tackles the challenging task of detecting propaganda techniques within text, specifically focusing on identifying both the span of text and the specific technique used. They propose a novel approach by reformulating the problem as a Machine Reading Comprehension (MRC) task, leveraging BERT for improved performance, and achieve a 3rd place ranking in the SemEval-2021 competition.

What Problem Does This Paper Address?

The proliferation of fake news and misinformation on social media presents a significant challenge. While traditional fake news detection often focuses on long-form content, this paper addresses the detection of propaganda techniques within memes, which are increasingly prevalent and can subtly influence audiences. The specific task (SemEval-2021 Task 6) involves not only identifying the presence of propaganda techniques but also pinpointing the exact text spans that employ them and classifying the technique itself. This is particularly difficult due to the potential for overlapping and nested spans, which standard sequence labeling models struggle with.

Methods

The authors initially explored a pipeline approach, first identifying text spans and then classifying the propaganda technique. However, this method proved insufficient for handling overlapping spans. To overcome this, they adopted a Machine Reading Comprehension (MRC) framework, inspired by question-answering systems.

Their MRC model uses BERT as the encoder. For each propaganda technique, they formulate a specific question. The input to BERT is a concatenation of this question and the text sequence. BERT then generates contextualized embeddings for each token.

Three specialized predictors are then employed:

  • START-Predictor: Identifies the probability of each token being the start of a propaganda span.
  • END-Predictor: Identifies the probability of each token being the end of a propaganda span.
  • SPAN-Predictor: A binary classifier that determines if a pair of start and end tokens constitutes a valid span, considering the probability margin.

The model is trained end-to-end by minimizing a combined loss function that includes cross-entropy losses for start, end, and span prediction. They also experimented with data augmentation techniques and explored different loss functions (Focal Loss and Asymmetric Loss) to address data imbalance.

Datasets

The primary dataset used is the SemEval-2021 Task 6 training dataset, which contains approximately 290 records. The authors also mention utilizing the PTC corpus for data augmentation.

Key Insights and Ideas

  • MRC for Span Detection: Reformulating the propaganda technique detection as a question-answering task using an MRC framework is a key innovation. This approach effectively handles the complexities of overlapping and nested spans, which are problematic for traditional sequence labeling.
  • Informative Queries: The effectiveness of the MRC framework hinges on crafting informative queries that guide the model to identify the relevant spans for each propaganda technique.
  • Data Augmentation and Loss Functions: Addressing data sparsity and imbalance is crucial. Data augmentation (specifically, enlarging the training dataset) and using Asymmetric Loss significantly improved performance, highlighting their importance in this task.
  • Negative Sampling Strategy: For the BERT-MRC Span Detection, sampling a specific number of negative examples (4 in their best configuration) was found to be beneficial, suggesting a nuanced approach to negative sampling is important.

Key Results

The authors achieved an F1 score of 0.3974 and secured 3rd place in the final evaluation phase of SemEval-2021 Task 6. Their experiments demonstrated that:

  • The MRC framework outperformed their initial pipeline approach.
  • Data augmentation, particularly enlarging the training dataset, led to a notable improvement in F1 score (from 0.19073 to 0.21053).
  • Asymmetric Loss provided a slight improvement over Focal Loss and Cross-Entropy for loss setup (0.20817 vs. 0.20453 vs. 0.19073).
  • The BERT-MRC model with 4 negative samples (BERT-MRCneg=4_DA) achieved the best result among their MRC variants (0.3779).

Limitations and Future Work

  • Data Sparsity: The limited size of the training dataset (290 records) is a significant limitation, which they attempted to mitigate with data augmentation.
  • Overlapping Span Handling: While the MRC approach is better than the pipeline, the paper implies that handling complex overlapping spans might still be an area for further improvement.
  • Noise in Data Augmentation: The paper notes that sampling too many negative examples during data augmentation can introduce noise and degrade performance, suggesting a need for careful selection.

Future work could involve exploring more advanced data augmentation techniques, investigating more sophisticated methods for handling complex span overlaps, and potentially incorporating multi-task learning to jointly predict spans and techniques more effectively.

Summary updated

BibTeX
@inproceedings{hou2021fpai,
  author = {Xiaolong Hou and Junsong Ren and Gang Rao and Lianxin Lian and Zhihao Ruan and Yang Mo and JIanping Shen},
  title = {{FPAI at SemEval-2021 Task 6: BERT-MRC for Propaganda Techniques Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure