Offensive Language Detection in Nepali Social Media

Nobal B. Niraula, Saurab Dulal, Diwa KoiralaACL 2021

TL;DR

This paper tackles the under-explored problem of detecting offensive language in Nepali social media. The authors introduce novel preprocessing techniques for the morphologically rich and multi-script Nepali language, release the first human-annotated dataset for Nepali offensive language detection, and establish baseline models for both coarse-grained and fine-grained classification.

What Problem Does This Paper Address?

The proliferation of user-generated content on social media has led to an increase in toxic and offensive language, including hate speech, personal attacks, and harassment. Detecting such content is crucial for user safety and maintaining healthy online discussions. While significant research has been dedicated to offensive language detection in resource-rich languages like English and German, low-resource languages like Nepali have been largely overlooked. This gap is problematic because linguistic nuances, cultural contexts, and script variations (e.g., Devanagari, Romanized Nepali, mixed scripts) make direct application of models trained on other languages ineffective. This paper aims to bridge this gap by characterizing offensive language in Nepali social media, addressing the unique challenges it presents, and providing foundational resources for future research.

Methods

The authors propose two novel preprocessing techniques to normalize Nepali social media text:

  • Dirghikaran (Prep_Dir): This method normalizes character variants that have the same sound in Devanagari script (e.g., mapping 'स' to 'श', 'व' to 'ब') to ensure consistency.
  • Romanization (Prep_Rom): This scheme transliterates Nepali words written in Devanagari script into their Romanized form, creating a uniform representation for mixed-script and transliterated text.

For feature extraction, they utilize word-based (unigrams, bigrams) and character-based (character trigrams) features, acknowledging Nepali's morphological richness. They experiment with various machine learning models, including Logistic Regression (LR), Support Vector Machines (SVM), Random Forests (RF), and Multilingual BERT (M-BERT). For classification, they implement both coarse-grained (offensive vs. non-offensive) and fine-grained (categorizing into sexist, racist, other-offensive) approaches. A unique two-step approach is used for fine-grained classification, where a coarse-grained classifier first identifies offensive content, and a second classifier then determines the specific category.

Datasets

The paper introduces the first human-labeled dataset for offensive language detection in Nepali social media. This dataset was created by collecting over 15,000 comments and posts from various platforms (Facebook, Twitter, YouTube, Nepali blogs, news portals). After manual annotation by two annotators, a total of 7462 records were labeled into four categories: SEXIST, RACIST, OTHER-OFFENSIVE, and NON-OFFENSIVE. The dataset is made publicly available. The training and testing splits are provided, with 5798 samples for training and 1450 for testing.

Key Insights and Ideas

  • Multilingualism and Script Diversity are Major Hurdles: Nepali social media text is not only noisy but also appears in Devanagari, Romanized Nepali, pure English, and mixed scripts, posing significant preprocessing challenges.
  • Morphological Richness Demands Character-Level Features: Nepali's complex morphology, where verbs can take numerous forms based on gender, number, and tense, makes character-based features particularly effective for capturing nuances missed by word-level analysis.
  • Romanization is Key for Normalization: The Romanization preprocessing technique proved highly effective, creating a unified text representation that significantly improved classification performance, especially when combined with Dirghikaran.
  • M-BERT Struggles with Low-Resource Languages: Despite its success in other languages, M-BERT did not perform as well as traditional models like Random Forests for Nepali, likely due to the limited Nepali data within its training corpus and its pre-training on Wikipedia, which differs from social media text.
  • Fine-grained Classification is Inherently Challenging: The performance for the "sexist" category was notably lower, attributed to the scarcity of training examples for this specific class, highlighting a common challenge in fine-grained offensive language detection.

Key Results

  • The Romanization + Dirghikaran preprocessing scheme combined with character features yielded the best results for binary classification, achieving an F1 score of 0.70 for the offensive class.
  • For coarse-grained classification, Random Forests and Logistic Regression models outperformed the baseline dictionary-based approach, achieving F1 scores of 0.73 and 0.72 for the offensive class, respectively.
  • In fine-grained classification, the Random Forest classifier achieved F1 scores of 0.87 for Non-Offensive, 0.71 for Other Offensive, 0.45 for Racist, and a very low 0.01 for Sexist.

Limitations and Future Work

  • Limitations: The authors acknowledge that the performance for the "sexist" category in fine-grained classification is low due to the limited number of training examples. They also note that M-BERT's performance was suboptimal, potentially due to its training on Wikipedia data rather than social media text and the under-representation of Nepali. The error analysis points to the difficulty of capturing world and contextual knowledge, as seen in the example differentiating "Thamel ma bhalu" from "jungle ma bhalu."
  • Future Work: The authors suggest future research could focus on detecting the targets of offensive comments (individuals, organizations, or groups). Additionally, leveraging offensive language datasets from other languages through translation and transfer learning is proposed as another promising direction. They also emphasize the importance of human review for flagged content, especially in cases of false positives, to avoid over-reliance on automated systems.

Summary updated

BibTeX
@inproceedings{niraula2021offensive,
  author = {Nobal B. Niraula and Saurab Dulal and Diwa Koirala},
  title = {{Offensive Language Detection in Nepali Social Media}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure