Detecting Social Media Manipulation in Low-Resource Languages

Samar Haider, Luca Luceri, Ashok Deb, Adam Badawy, Nanyun Peng, Emilio Ferrara

Introduction

Research into social media manipulation—encompassing bots, trolls, and disinformation—has predominantly focused on high-resource languages like English and Spanish. However, malicious actors operate globally, often targeting regions where local languages have fewer digital resources or limited training data for AI models. This creates a significant vulnerability in detecting coordinated inauthentic behavior in "low-resource" linguistic environments.

Detecting these accounts is challenging due to data scarcity, linguistic complexity (especially in morphologically rich languages), and the rapidly evolving tactics of malicious actors. This study focuses on Tagalog, a language that was surprisingly prevalent in Twitter data during the 2016 US Presidential election but remains underserved by standard NLP tools.

Methodology

The researchers developed a framework designed to leverage knowledge from high-resource languages to improve detection in low-resource ones. The pipeline combines two primary techniques:

  1. FastText Embeddings: Unlike models that treat words as atomic units, FastText represents words as bags of character n-grams. This subword-level understanding is vital for informal social media text, allowing the model to process rare words or misspellings based on their components.
  2. Cross-Lingual Alignment (MUSE): Using the Multilingual Unsupervised and Supervised Embeddings (MUSE) framework, the researchers aligned the latent spaces of English and Tagalog models. By mapping Tagalog words into the same mathematical space as English words using a bilingual dictionary, the model can "transfer" its understanding of malicious patterns identified in English to the Tagalog context.

Data and Experimental Setup

The study utilized a massive dataset of approximately 42 million tweets from the 2016 US Presidential election. Within this, a significant Tagalog subset was identified. Twitter's own enforcement actions provided a "ground truth" label: approximately 31% of the Tagalog-posting accounts were eventually suspended for malicious activity.

Key Results

Performance and Data Efficiency

The transfer learning approach (FastText + MUSE) demonstrated superior performance, particularly in scenarios where training data was extremely scarce. When restricted to just 10% of the available Tagalog training data, the transfer learning model achieved nearly three times the recall (0.147) of a monolingual FastText model (0.051).

Comparison of F1 scores between monolingual and transfer learning models across varying training data sizes.

Overall, the FastText + MUSE approach achieved an F1-score of 0.347, significantly outperforming more complex models like multilingual BERT (0.215). The authors attribute BERT's lower performance to its training on formal text (Wikipedia), which struggles with the slang-heavy, informal nature of social media manipulation.

Visualizing Account Separation

To confirm that the model successfully captured linguistic markers of manipulation, the authors used UMAP for dimensionality reduction. The resulting visualization shows a clear separation between the embeddings of suspended and non-suspended accounts, validating the model's ability to distinguish malicious intent through language.

UMAP visualization of account embeddings, demonstrating the separation between suspended (red) and non-suspended (blue) accounts.

Discussion and Limitations

A notable finding was the "conservative" bias of the models, which exhibited higher precision than recall. In a real-world moderation context, this is often preferred, as it minimizes the risk of "false positives"—wrongfully suspending legitimate users.

While successful, the study relies on account suspension as a proxy for maliciousness, which can occasionally include general spam or copyright violations. Future work aims to incorporate network features (follower/following relationships) and metadata to complement the current text-only analysis, and to test the framework on even more resource-starved languages where bilingual dictionaries are unavailable.

Summary updated

Abstract

Social media have been deliberately used for malicious purposes, including political manipulation and disinformation. Most research focuses on high-resource languages. However, malicious actors share content across countries and languages, including low-resource ones. Here, we investigate whether and to what extent malicious actors can be detected in low-resource language settings. We discovered that a high number of accounts posting in Tagalog were suspended as part of Twitter's crackdown on interference operations after the 2016 US Presidential election. By combining text embedding and transfer learning, our framework can detect, with promising accuracy, malicious users posting in Tagalog without any prior knowledge or training on malicious content in that language. We first learn an embedding model for each language, namely a high-resource language (English) and a low-resource one (Tagalog), independently. Then, we learn a mapping between the two latent spaces to transfer the detection model. We demonstrate that the proposed approach significantly outperforms state-of-the-art models, including BERT, and yields marked advantages in settings with very limited training data -- the norm when dealing with detecting malicious activity in online platforms.
BibTeX
@inproceedings{haider2020detecting,
  author = {Samar Haider and Luca Luceri and Ashok Deb and Adam Badawy and Nanyun Peng and Emilio Ferrara},
  title = {{Detecting Social Media Manipulation in Low-Resource Languages}},
  booktitle = {The Web Conference},
  year = {2020},
  doi = {10.1145/3543873.3587615},
}
Figure