Overview of the Shared Task on Multimodal Hate Speech Detection in Dravidian languages: DravidianLangTech@NAACL 2025

Jyothish Lal G, Premjith B, Bharathi Raja Chakravarthi, Saranya Rajiakodi, Bharathi B, Rajeswari Natarajan, Ratnavel RajalakshmiNAACL 2025

TL;DR

This paper introduces DravidianLangTech@NAACL 2025, a shared task focused on detecting hate speech in multimodal (speech + text) content from low-resource Dravidian languages (Malayalam, Tamil, Telugu). The task highlights the effectiveness of transformer-based models and multimodal approaches, with top performers achieving significant improvements, especially in Malayalam and Tamil.

What Problem Does This Paper Address?

Hate speech on social media poses a significant threat to mental health, social harmony, and online safety. While detecting hate speech in text has seen progress, its detection in spoken language and its combination with text, particularly in low-resource languages, remains a challenge due to a lack of suitable datasets and multimodal approaches. This paper addresses this gap by organizing a shared task and presenting a new multimodal dataset for hate speech detection in three Dravidian languages: Malayalam, Tamil, and Telugu.

Methods

The shared task involved participants developing models to classify hate speech in multimodal content. The primary methods employed by participants included:

  • Transformer-based Models: Leading teams extensively utilized fine-tuned BERT models, including language-specific variants and multilingual models like XLM-ROBERTa, for text processing.
  • Multimodal Fusion: Various techniques were used to combine speech and text features. Common approaches included:
  • Late Fusion: Concatenating features extracted from separate text and speech models.
  • Attention Mechanisms: Employed to weigh the importance of different features.
  • Early Fusion: Integrating features at an earlier stage of processing.
  • Speech Feature Extraction: Techniques like spectrograms, MFCCs, and spectral contrast were used to extract acoustic features. Pre-trained speech models like wav2vec2 and whisper were also utilized.
  • Handling Class Imbalance: Techniques such as SMOTE (Synthetic Minority Over-sampling Technique) were used to address the imbalance in hate speech vs. non-hate speech data.

Datasets

The dataset for this shared task was curated from YouTube videos on channels with over 50,000 subscribers. The selection focused on topics likely to generate controversial discussions, such as the Ram Mandir inauguration and defamation of public figures. The dataset includes both speech and corresponding text transcripts. The paper categorizes hate speech into four types based on YouTube's policy: Gender-based hate speech (G), Political hate speech (P), Religious hate speech (R), and Personal defamation (C), along with a Non-hate (N) class.

Key Insights and Ideas

  • Multimodal Advantage: The results strongly suggest that combining speech and text information significantly improves hate speech detection, especially for low-resource languages where individual modalities might be less informative.
  • Transformer Dominance: Fine-tuned transformer models, particularly those adapted to specific languages or multilingual contexts, proved highly effective for text-based hate speech detection.
  • Low-Resource Challenges: While progress was made, Telugu consistently showed lower performance compared to Malayalam and Tamil, indicating that language-specific nuances and data availability remain critical challenges for low-resource languages.
  • Data Curation Importance: The paper emphasizes the value of carefully curated datasets that capture context and intent, rather than relying solely on keyword matching, for identifying nuanced hate speech.

Key Results

  • Top Performers: Team SSNTrio achieved the highest macro F1 scores in Malayalam (0.7511) and Tamil (0.7332). Team lowes secured the best F1 score in Telugu (0.3817).
  • Malayalam and Tamil Success: The top teams demonstrated strong performance in Malayalam and Tamil, indicating the effectiveness of the developed multimodal approaches for these languages.
  • Telugu Performance Gap: The macro F1 scores for Telugu were generally lower across all teams, highlighting the greater difficulty in detecting hate speech in this language within the given dataset. For instance, the top score in Telugu was 0.3817, significantly lower than the top scores in Malayalam and Tamil.

Limitations and Future Work

  • Limitations:
    • The paper notes that Telugu posed greater challenges, suggesting potential limitations in the dataset or the complexity of hate speech in that language.
  • While not explicitly stated as a limitation, the performance gap between languages implies that current multimodal approaches may not be equally effective across all Dravidian languages without further adaptation or data.
  • Future Work:
    • The paper suggests that the creation of a comprehensive, multiclass, multimodal dataset enriches resources for future research.
  • The results underscore the potential of combining textual and vocal features for robust hate speech detection, paving the way for more inclusive and accurate models. This implies continued research into novel multimodal fusion techniques and language-specific adaptations.

Summary updated

BibTeX
@inproceedings{g2025overview,
  author = {Jyothish Lal G and Premjith B and Bharathi Raja Chakravarthi and Saranya Rajiakodi and Bharathi B and Rajeswari Natarajan and Ratnavel Rajalakshmi},
  title = {{Overview of the Shared Task on Multimodal Hate Speech Detection in Dravidian languages: DravidianLangTech@NAACL 2025}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure