Kb at SemEval-2023 Task 3: On Multitask Hierarchical BERT Base Neural Network for Multi-label Persuasion Techniques Detection

Katarzyna Baraniak, M SydowACL 2023

TL;DR

This paper introduces a novel multitask hierarchical BERT-based neural network for detecting persuasion techniques in news articles. By jointly learning to identify the specific spans of text containing persuasion and classifying the techniques used, the model achieves improved performance, especially in low-data scenarios, and offers a more efficient approach than separate models.

What Problem Does This Paper Address?

News articles, while aiming for objectivity, often employ subtle forms of persuasion that can influence readers, particularly those with lower political knowledge. Automatically detecting these persuasion techniques is crucial for media analysis and fostering a more informed public. The SemEval-2023 Task 3 specifically targets this by requiring the identification of persuasion techniques within paragraphs of news articles across multiple languages. The challenge lies in accurately pinpointing the exact text segments (spans) that contain persuasion and then classifying them into one or more of 23 predefined categories, which are themselves imbalanced.

Methods

The authors propose a multitask hierarchical neural network built upon a pre-trained BERT base model. The core innovation lies in the joint learning of two tasks:

  1. Span Identification: This auxiliary task uses a linear layer to classify each token in a paragraph as either belonging to a persuasion span ('I') or not ('O'). This is framed as a sequence tagging problem.
  2. Multi-label Persuasion Technique Classification: This is the main task. The model leverages the output of the span identification task. Specifically, it uses the BERT embedding of the first token of an identified persuasion span as input to a second linear layer. This layer, followed by a sigmoid activation, predicts the presence of one or more persuasion techniques for that span. If no span is identified, it defaults to using the embedding of the first token of the entire paragraph.

This hierarchical structure allows the model to learn from both tasks simultaneously, with the span identification guiding the technique classification. The loss function is a weighted sum of the cross-entropy loss for span identification and a weighted binary cross-entropy loss for multi-label classification, with weights adjusted to account for class imbalance.

Datasets

The primary dataset used is the SemEval-2023 Task 3 dataset, which includes news articles in English, French, German, Italian, Polish, and Russian, with training and development sets. For Spanish, Greek, and Georgian, only test data was available. For these languages, the authors translated articles from Polish to generate training data, acknowledging that this translation process might not perfectly preserve persuasion nuances.

Key Insights and Ideas

  • Auxiliary Span Detection Boosts Performance: The paper demonstrates that adding span identification as an auxiliary task significantly improves the performance of the main persuasion technique classification task. This suggests that explicitly identifying where persuasion occurs is a crucial precursor to classifying what persuasion is being used.
  • Leveraging Span Start Embeddings: A key finding is that using the BERT embedding of the first token of an identified span for classification yields better results than using the embedding of the first token of the entire paragraph. This highlights the importance of focusing on the most salient part of the persuasive text.
  • Efficiency of Single Network: By combining span detection and classification into a single multitask network, the authors achieve a more data-efficient solution. This is particularly beneficial when dealing with limited training data, as the network can learn from both tasks simultaneously.
  • Translation Challenges: While translation was used to augment data for some languages, the results suggest that it doesn't fully compensate for the lack of native training data, indicating that persuasion techniques might be language-specific or lost in translation.

Key Results

  • The proposed multitask hierarchical BERT model consistently outperforms the baseline across all tested languages on the official test sets.
  • For languages with full training data (English, Polish, French, Italian, Russian, German), the model achieves significantly better micro-F1 scores compared to the baseline. For example, in English, the model achieved a micro-F1 of 0.06022, while the baseline was 0.19517 (note: the paper mentions a potential issue with the English test file upload, and the * rank is calculated based on the official leaderboard).
  • Even for languages with only translated data (Spanish, Greek, Georgian), the model still performs better than the baseline, though with considerably lower scores, underscoring the limitations of translation.
  • The standard deviation of micro-F1 scores across multiple runs on the dev set indicates stable performance.

Limitations and Future Work

  • Translation Quality: The performance for languages with translated data is significantly lower, suggesting that translation may not be an ideal substitute for native training data. Future work could explore more sophisticated cross-lingual transfer learning methods.
  • Context Dependency: The error analysis reveals that some persuasion techniques, like "Red Herring" or "Conversation Killer," are difficult to detect without broader contextual information beyond a single paragraph. The authors suggest that incorporating more context could improve accuracy.
  • Ensemble Models: The diversity and difficulty of persuasion techniques lead the authors to believe that ensemble models might offer improved results.
  • Full Span Utilization: Currently, if multiple spans are detected, the model primarily uses the first one for classification. Future work could explore methods to incorporate information from all detected spans.
  • More Powerful Models: The authors suggest investigating other language-specific BERT models or more advanced architectures for further improvements.

Summary updated

BibTeX
@inproceedings{baraniak2023kb,
  author = {Katarzyna Baraniak and M Sydow},
  title = {{Kb at SemEval-2023 Task 3: On Multitask Hierarchical BERT Base Neural Network for Multi-label Persuasion Techniques Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure