Abstract
Bias is ubiquitous in most online sources of natural language, from news media to social networks. Given the steady shift in news consumption behavior from traditional outlets to online sources, the automatic detection of propaganda, in which information is shaped to purposefully foster a predetermined agenda, is an increasingly crucial task. To this goal, we explore the task of sentence-level propaganda detection, and experiment with both handcrafted features and learned dense semantic representations. We also experiment with random undersampling of the majority class (nonpropaganda) to curb the influence of class distribution on the system's performance, leading to marked improvements on the minority class (propaganda). Our best performing system uses pre-trained ELMo word embeddings, followed by a bidirectional LSTM and an attention layer. We have submitted a 5-model ensemble of our best performing system to the NLP4IF shared task on sentence-level propaganda detection (team LIACC), achieving rank 10 among 25 participants, with 59.5 F1-score.
Introduction
As news consumption increasingly shifts toward online platforms and social media, the spread of propaganda—information strategically shaped to promote a specific agenda—presents a significant societal challenge. Unlike "fake news," which often relies on factual falsehoods, propaganda is frequently characterized by subtle linguistic techniques and persuasive framing. This study addresses the challenge of Sentence-Level Propaganda Detection (SLC) by evaluating two primary approaches: traditional handcrafted stylometric features and modern dense semantic representations derived from deep learning.
Methodology
The research evaluates two distinct pipelines for feature representation and classification:
Handcrafted Stylometric Features
The first approach utilizes linguistically-inspired features, including sentence and word length, punctuation frequency, and vocabulary diversity (type-token ratio). Additionally, the top 50 most frequent n-grams were extracted using TF-IDF. These features were used to train a Gradient Boosted Trees (GBT) model, providing an interpretable baseline based on traditional NLP techniques.
Deep Learning and Attention Mechanisms
The second approach leverages contextualized word representations. The authors utilized ELMo (Embeddings from Language Models) to generate 3072-dimensional vectors for each token. These embeddings were processed through a Bidirectional LSTM (BiLSTM) to capture sequential dependencies.
To further refine the model, an Attention-based BiLSTM (ABL) was developed. This architecture, shown in Figure 1, incorporates an attention layer that allows the model to assign varying weights to specific tokens, effectively identifying which words contribute most to a propaganda classification.
Addressing Class Imbalance
A significant hurdle in propaganda detection is the inherent class imbalance of news datasets. In the NLP4IF 2019 dataset, propaganda sentences are heavily outnumbered by non-propaganda sentences. To mitigate this, the authors employed random undersampling of the majority class during training to ensure the models were exposed to a balanced distribution of examples.
Experimental Results
The models were evaluated on the NLP4IF 2019 shared task dataset, which consists of 16,965 sentences from 350 articles. The results, summarized in Figure 2, highlight two critical findings: the superiority of deep learning over handcrafted features and the vital role of data balancing.
Deep learning models (BiLSTM and ABL) significantly outperformed the GBT model. On balanced data, the ABL model achieved the highest F1-score of 75.0. Furthermore, training on imbalanced data led to a dramatic drop in performance across all architectures; for instance, the GBT model's F1-score fell from 67.7 to 32.6 when the training set was not balanced.
Analysis of the attention weights revealed that the ABL model often focused on specific opinion-inducing words (e.g., "unprecedented", "horrify") or specific verb conjugations when identifying propaganda. Interestingly, propaganda detection tended to trigger a concentrated attention spike on a single word, whereas non-propaganda sentences resulted in a more diffused attention distribution across the sentence.
Conclusion
This study demonstrates that while handcrafted stylometric features provide a baseline, dense semantic representations combined with attention mechanisms offer a more robust solution for detecting the subtle cues of propaganda. The success of the ABL model, which ranked 10th in the NLP4IF competition with an ensemble F1-score of 59.5, underscores the importance of context and class-balancing strategies. Future work will explore the fusion of handcrafted features with neural embeddings and the use of more diverse classifier ensembles to further improve detection accuracy.
Summary updated