Trio Innovators @ DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian Languages

Radha N, Swathika R, Farha Afreen I, Annu G, Apoorva ANAACL 2025

TL;DR

This paper tackles the challenging problem of detecting hate speech in Dravidian languages (Tamil, Telugu, Malayalam) by combining audio and text data. They developed a multimodal framework that achieves a macro F1-score of 0.76, significantly outperforming unimodal approaches and paving the way for more robust hate speech detection in under-resourced languages.

What Problem Does This Paper Address?

Hate speech is a pervasive issue on social media, and its detection is particularly difficult in languages with limited linguistic resources, like Tamil, Telugu, and Malayalam. Traditional text-only hate speech detection models often miss crucial nuances conveyed through audio, such as tone, sarcasm, and emotion, which are vital for accurate identification. This paper addresses this gap by proposing a multimodal approach that leverages both audio and text to improve hate speech detection accuracy in these under-resourced Dravidian languages.

Methods

The authors propose a multimodal framework that integrates advanced techniques for audio and text feature extraction, followed by feature alignment, fusion, and classification.

  • Audio Feature Extraction: They utilize the Wav2Vec 2.0 model to extract speech features. This model captures acoustic properties like tone, pitch, intensity, and speech rhythm. Additional features like MFCCs, spectral contrast, zero-crossing rate, and log Mel spectrogram are also extracted to enhance the representation of audio data.
  • Text Feature Extraction: For text, they employ XLM-Roberta, a multilingual transformer model, to generate embeddings that capture semantic meaning, context, and intent. The [CLS] token embedding is used as the sentence-level representation. The text preprocessing involves cleaning punctuation and unwanted characters.
  • Feature Alignment: Audio and text features are aligned using file names to ensure consistency for each data instance.
  • Feature Fusion: The aligned audio and text feature vectors are horizontally concatenated to create a unified feature representation. This fused representation allows the model to leverage both acoustic and semantic information.
  • Handling Class Imbalance: To address the issue of imbalanced datasets (where certain hate speech categories are underrepresented), they use the Synthetic Minority Over-sampling Technique (SMOTE). SMOTE generates synthetic samples for minority classes to create a more balanced dataset.
  • Classification Model: The XGBoost model is used for classification. Known for its robustness with imbalanced datasets, XGBoost is employed to predict the hate speech category based on the fused feature vector.

Datasets

The paper introduces a novel, real-world dataset specifically curated for multimodal hate speech detection in Tamil, Telugu, and Malayalam.

  • Dataset Size: The dataset comprises 300 samples, equally divided among the three languages.
  • Categories: It is categorized into five classes: Gender-based Hate (G), Political Hate (P), Religious Hate (R), Personal Defamation (C), and Non-Hate Speech (NH).
  • Structure: The dataset includes both text and audio for each sample.
  • Distribution: The training set has 407 Tamil, 440 Telugu, and 706 Malayalam samples. Validation sets are also provided, and the test set is balanced with 50 samples per language.
  • Challenges: The dataset presents challenges such as class imbalance (Personal Defamation is underrepresented) and code-mixing (mixing of native and English scripts), reflecting real-world social media usage.

Key Insights and Ideas

  • Multimodality is Crucial for Dravidian Languages: The study strongly emphasizes that combining audio and text is essential for accurate hate speech detection in Dravidian languages. Audio cues like tone and emotion, often missed by text-only models, provide critical context.
  • Complementary Roles of Modalities: The results show that text-only models had a lower macro F1-score (0.68), while audio-only models performed better (0.72). However, the fused multimodal model achieved the highest score (0.76), demonstrating that each modality contributes uniquely and their combination is synergistic.
  • SMOTE Effectively Addresses Imbalance: The application of SMOTE significantly improved recall for minority classes, highlighting its effectiveness in mitigating class imbalance issues in hate speech detection.
  • XGBoost's Robustness: The choice of XGBoost proved effective in handling the imbalanced dataset and learning complex patterns from the fused multimodal features.

Key Results

The proposed multimodal hate speech detection system achieved significant results:

  • Overall Performance: The model achieved a macro F1-score of 0.76 and an accuracy of approximately 85%.
  • Comparison of Feature Types:
    • Malayalam: Macro F1-score of 0.40 (audio-text dataset) vs. 0.17 (audio dataset).
  • Tamil: Macro F1-score of 0.35 (audio-text dataset) vs. 0.17 (audio dataset).
  • Telugu: Macro F1-score of 0.44 (audio-text dataset) vs. 0.19 (audio dataset).
  • These results clearly indicate the superiority of the combined audio-text approach over unimodal methods.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Limitations:
    • Small and Limited Dataset: The dataset is limited to only three Dravidian languages, restricting generalization to other Dravidian languages and diverse linguistic contexts.
  • Reliance on Both Modalities: The model's performance is dependent on the availability of both audio and text, which might not always be practical in real-world scenarios.
  • Class Imbalance: Despite SMOTE, the "Personal Defamation" category remains underrepresented, and SMOTE might not fully capture real-world data distribution.
  • Code-Mixing Complexity: Detecting hate speech with code-mixing (native language + English) poses a challenge due to the need for understanding both languages.
  • Lack of Visual Data: The absence of video data means the model cannot leverage visual cues like facial expressions or gestures, which could enhance detection.
  • Bias Assessment: The study did not assess potential biases in the model.
  • Future Work:
    • Dataset Expansion: Expanding the dataset to include more Dravidian languages and a larger number of samples.
  • Incorporating Video Data: Adding video modality to provide richer contextual information.
  • Improving Class Imbalance Handling: Exploring more advanced techniques to address class imbalance.
  • Bias Mitigation: Investigating and mitigating potential biases in the model.
  • Domain-Specific Fine-tuning: Refining models with contextual embeddings and domain-specific fine-tuning.

Summary updated

BibTeX
@inproceedings{n2025trio,
  author = {Radha N and Swathika R and Farha Afreen I and Annu G and Apoorva A},
  title = {{Trio Innovators @ DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian Languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure