DLRG@DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian Languages

Ratnavel Rajalakshmi, Ramesh Kannan, Meetesh Saini, Bitan MallikNAACL 2025

TL;DR

This paper tackles the critical challenge of detecting hate speech in Tamil, a low-resource Dravidian language, by proposing a novel multimodal approach that combines audio and textual analysis. Their system leverages Whisper for accurate speech-to-text transcription and Indic-BERT for nuanced textual understanding, achieving a respectable F1 score of 0.25 in a shared task.

What Problem Does This Paper Address?

Hate speech is a pervasive online problem that fuels discrimination and violence. While significant research has focused on detecting hate speech in high-resource languages like English, Dravidian languages, such as Tamil, remain under-addressed. This is particularly challenging due to their unique linguistic structures and the multimodal nature of hate speech, which can manifest not only in text but also in audio. Existing methods often struggle with the nuances of these languages and the integration of different modalities.

Methods

The authors propose a multimodal system that integrates two key components:

  • Whisper for Audio-to-Text Transcription: They utilize the Whisper model, specifically a Tamil-configured version (vasista22/whisper-tamil-medium), to transcribe spoken Tamil audio into text. This is crucial for capturing hate speech that might be present in audio but not in accompanying text.
  • Indic-BERT for Textual Classification: For the textual analysis, they employ Indic-BERT, a language model pre-trained on a diverse set of Indian languages. The authors argue that Indic-BERT is superior to generic multilingual models like mBERT and XLM-R for Tamil due to its ability to capture specific linguistic variations.

The system's architecture involves feeding raw audio into Whisper for transcription. The resulting text is then preprocessed (URLs and non-alphanumeric characters removed, tokenized to a max length of 128) and fed into the Indic-BERT classifier. This approach aims to leverage the strengths of both speech recognition and language understanding models for a more robust hate speech detection system.

For comparison, they also developed a unimodal Wav2Vec2-based model that solely focuses on audio features. This model preprocesses raw audio, passes it through the Wav2Vec2 model to obtain contextualized hidden states, performs mean pooling, and then uses a dropout layer and a fully connected layer for classification.

Datasets

The primary dataset used is the Multimodal Hate Speech Tamil dataset. This dataset contains:

  • Audio data: YouTube videos with diverse speakers.
  • Textual data: Transcriptions of the spoken content.
  • Labels: Five categories: N (Non-hate speech), G (Gender-based hate speech), P (Political hate speech), R (Religious hate speech), and C (Communal hate speech).

The dataset consists of 514 samples, with WAV files and corresponding transcriptions. A separate test dataset of 50 audio and text samples was provided for evaluation, though without labels, requiring researchers to submit predictions.

Key Insights and Ideas

  • Multimodality is Key for Dravidian Languages: The paper strongly advocates for multimodal approaches, recognizing that hate speech in languages like Tamil can be conveyed through both spoken and written forms, and integrating these modalities is essential for comprehensive detection.
  • Indic-BERT's Advantage: The authors highlight the benefit of using Indic-BERT over more general multilingual models for Tamil, suggesting its specialized pre-training captures language-specific nuances more effectively.
  • Whisper's Efficacy for Tamil Transcription: The use of Whisper demonstrates its capability in handling Tamil speech transcription, a critical step for enabling subsequent textual analysis.
  • Wav2Vec2 as a Strong Unimodal Baseline: The unimodal Wav2Vec2 model, focusing solely on audio features, achieved a higher F1 score than the multimodal approach, suggesting that audio features alone can be quite informative, or that the audio-text integration in their multimodal model needs further refinement.

Key Results

  • The multimodal Indic-BERT and Whisper-based system achieved an F1 score of 0.25 on the Tamil Hate Speech Detection task.
  • The unimodal Wav2Vec2-based model achieved a higher F1 score of 0.35 and an accuracy of 51%.
  • The proposed multimodal approach ranked 10th position in the shared task on Multimodal Hate Speech Detection in Dravidian languages at the NAACL 2025 Workshop DravidianLangTech.

Limitations and Future Work

Limitations:

  • The multimodal system's F1 score (0.25) was lower than the unimodal audio-only approach (0.35), indicating potential areas for improvement in how audio and text are integrated.
  • Whisper, while effective, is noted to have occasional transcription errors, which can impact downstream classification.

Future Work:

The authors suggest several promising directions:

  • Ensembling: Combining multiple models to leverage their diverse strengths.
  • Dataset Improvements: Further enhancing the dataset for better representation and coverage.
  • Optimized Audio-Text Integration: Developing more sophisticated methods to fuse audio and textual information.
  • Exploring Other Architectures: Investigating alternative models and feature extraction techniques.
  • Addressing Low-Resource Challenges: Continuing to focus on improving hate speech detection in under-resourced languages.

Summary updated

BibTeX
@inproceedings{rajalakshmi2025dlrg,
  author = {Ratnavel Rajalakshmi and Ramesh Kannan and Meetesh Saini and Bitan Mallik},
  title = {{DLRG@DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure