3218IR at SemEval-2020 Task 11: Conv1D and Word Embedding in Propaganda Span Identification at News Articles

Dimas Sony Dewantara, Indra Budi, Muhammad Okky IbrohimCOLING 2020

TL;DR

This paper tackles the challenging task of identifying propaganda spans within news articles using a 1-Dimensional Convolutional Neural Network (Conv1D) combined with GloVe word embeddings. While achieving a modest F1 score of 0.24 on the development set, the research highlights the potential for performance improvements by fine-tuning hyper-parameters, particularly those related to the Conv1D architecture.

What Problem Does This Paper Address?

In today's information landscape, distinguishing between genuine news and propaganda is increasingly difficult. Propaganda aims to subtly influence readers by shaping information to promote a specific agenda. Identifying these propaganda techniques at a granular level – specifically, pinpointing the exact text fragments that constitute propaganda within a sentence or paragraph – is a crucial but complex task. This paper addresses SemEval-2020 Task 11, Subtask 1, which focuses on this "propaganda span identification" problem in news articles. The challenge lies in accurately detecting these biased fragments without being overly sensitive or missing subtle instances.

Methods

The authors propose a deep learning approach utilizing a 1-Dimensional Convolutional Neural Network (Conv1D) as the core architecture, augmented with GloVe word embeddings.

  • Conv1D: This is a type of Convolutional Neural Network (CNN) well-suited for sequential data like text. It works by applying convolutional filters to extract local features (patterns) from word sequences. The paper explores variations in the number of convolutional layers, filter sizes, kernel sizes, and pooling sizes to optimize performance.
  • GloVe Word Embeddings: Instead of using simple one-hot encodings for words, the model leverages pre-trained GloVe embeddings. GloVe represents words as dense vectors in a low-dimensional space, capturing semantic relationships between words. This is known to improve NLP task performance by providing a more expressive input representation. The authors specifically used a 200-dimensional GloVe variant.
  • Model Architecture: The proposed model consists of an embedding layer (for GloVe), followed by one or more Conv1D layers, MaxPooling1D layers, a Dropout layer (to prevent overfitting), and finally a Dense layer with a sigmoid activation function for binary classification (propaganda vs. non-propaganda).

Datasets

The dataset used in this research is provided by SemEval-2020 Task 11. It comprises approximately 550 English news articles. These articles are annotated with fragments containing one of 18 propaganda techniques. The dataset is provided in two files: one containing the articles with sentences separated by rows, and another containing labels with article IDs, begin, and end offsets for the propaganda fragments. The authors split this data into a training set (13338 pairs of sentences and labels) and a validation set (3335 pairs).

Key Insights and Ideas

  • Conv1D's Potential for Text: The research reinforces the effectiveness of Conv1D for text classification tasks, particularly at the fragment level, by its ability to extract salient features from word sequences.
  • Hyper-parameter Tuning is Crucial: The paper strongly emphasizes that hyper-parameter tuning, especially for the Conv1D layers (number of layers, filter counts, kernel sizes, and pooling sizes), is critical for achieving optimal performance. Their best model emerged from a systematic exploration of these parameters.
  • GloVe Enhances Representation: The use of GloVe embeddings, as opposed to simpler word representations, likely contributed to the model's ability to capture nuanced linguistic patterns indicative of propaganda.

Key Results

The best-performing model achieved an F1-score of 0.24 on the development set and 0.23 on the test set. This placed them 32nd out of 36 participants in the SemEval-2020 Task 11. While these scores are modest, the paper highlights that the best scenario (Sce 16) involved 3 convolutional layers with a specific pattern of increasing kernel sizes and decreasing pooling sizes across layers.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Hyper-parameter Exploration: They suggest that further performance improvements could be achieved by exploring a wider range of hyper-parameter combinations that were not tested in this study.
  • Static Parameters: Some parameters were kept static (e.g., learning rate, dropout value, batch size, epochs). Investigating these could also lead to better results.

Future work could involve exploring more complex model architectures, different word embedding techniques, or incorporating external knowledge sources to better detect propaganda. The current results, while not top-tier, demonstrate a viable approach and provide a solid baseline for future research in this important area.

Summary updated

BibTeX
@inproceedings{dewantara2020ir,
  author = {Dimas Sony Dewantara and Indra Budi and Muhammad Okky Ibrohim},
  title = {{3218IR at SemEval-2020 Task 11: Conv1D and Word Embedding in Propaganda Span Identification at News Articles}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure