Abstract
Detecting hate speech in low-resource languages remains a critical challenge for online safety. This paper presents a robust system for identifying hate speech and its targets in Devanagari-scripted languages, specifically Hindi and Nepali. By leveraging Parameter-Efficient Fine-Tuning (PEFT) via LoRA, the authors demonstrate a cost-effective approach to adapting Large Language Models (LLMs) for these specific linguistic contexts. Among the evaluated models, Nemo-Instruct-2407 emerged as the top performer, achieving high overall accuracy despite significant dataset imbalances.
Introduction
While hate speech detection has seen substantial progress in high-resource languages like English, low-resource languages—particularly those using the Devanagari script—are often overlooked. This gap is concerning, as online hostility in these regions can quickly translate into real-world harm. The primary technical obstacles include a lack of high-quality annotated datasets and the high computational costs associated with full fine-tuning of modern LLMs.
Methodology
To address computational constraints, the researchers employed Parameter-Efficient Fine-Tuning (PEFT) using the LoRA (Low-Rank Adaptation) technique. LoRA freezes the original model weights and introduces trainable low-rank matrices into the attention layers, significantly reducing the memory footprint and computational requirements.
The models were further optimized using 4-bit quantization via the Unsloth library, enabling training on consumer-grade hardware like a single 16GB NVIDIA T4 GPU. The study benchmarked four prominent models: Llama-3.1, Nemo-Instruct-2407, Qwen2.5, and Phi-3-medium.
Dataset and Task Description
The study utilized data from the CHiPSAL 2025 shared task, covering two primary objectives:
- Task 1 (Detection): Binary classification (Hate vs. Not Hate). The dataset is heavily imbalanced, with "Not Hate" instances outnumbering "Hate" instances by nearly 8 to 1.
- Task 2 (Target Identification): Multi-class classification identifying the target as an Individual, Organization, or Community.
Results and Discussion
The Nemo-Instruct-2407 model consistently outperformed other architectures across both tasks. In the detection task, it achieved an accuracy of 90.75% and a weighted F1 score of 90.05%.
The Challenge of Class Imbalance
A significant finding was the impact of data distribution on model reliability. While overall accuracy was high, the models struggled with the minority "Hate" class. As shown in the confusion matrix below, a substantial portion of actual hate speech instances were misclassified as "no hate."
Target identification proved even more complex, with F1 scores dropping to approximately 71%. Distinguishing between "Organization" and "Community" targets requires deep sociopolitical context that remains difficult for models to capture from limited training samples.
Conclusion
This research demonstrates that PEFT-based LLMs are a viable and efficient solution for hate speech detection in low-resource Devanagari languages. Nemo-Instruct-2407 showed particular promise, suggesting its base pre-training is well-aligned with these scripts. Future work will focus on synthetic data generation to address class imbalances and the exploration of ensemble methods to further improve detection robustness.
Summary updated