SemanticCuetSync@DravidianLangTech 2025: Multimodal Fusion for Hate Speech Detection - A Transformer Based Approach with Cross-Modal Attention

Md. Sajjad Hossain, Symom Hossain Shohan, Ashraful Islam Paran, Jawad Hossain, Mohammed Moshiul HoqueNAACL 2025

TL;DR

This paper tackles the challenging problem of detecting hate speech in Telugu, a low-resource language, by proposing a multimodal approach that fuses text and audio information. They developed a novel cross-modal attention mechanism to effectively combine these modalities, achieving a respectable 3rd place in the DravidianLangTech 2025 shared task.

What Problem Does This Paper Address?

The proliferation of social media has unfortunately led to a rapid spread of hate speech, making content moderation a critical but difficult task. This challenge is amplified in low-resource languages like Telugu, where research on hate speech detection is scarce. While previous work has explored unimodal approaches (text or audio alone), effectively detecting hate speech often requires understanding nuances present in both spoken and written communication. This paper aims to bridge this gap by developing a multimodal hate speech detection system for Telugu, leveraging the power of transformers and cross-modal attention.

Methods

The authors explored a variety of machine learning (ML) and deep learning (DL) models, with a particular focus on transformer-based architectures. For feature extraction, they employed several transformer models for text, including BERT, TeluguBERT, DistilBERT, and HateSpeechBERT, as well as Whisper (for audio) and Mel-frequency Cepstral Coefficients (MFCCs). The core of their novel approach lies in the cross-modal attention mechanism. This technique allows the model to learn how to weigh and integrate information from both the text and audio modalities, enabling a more comprehensive understanding of potential hate speech. After feature extraction and cross-modal attention, a simple concatenation (early fusion) was used to combine the fused features for the final classification.

Datasets

The study utilized a multimodal hate speech dataset created by Anilkumar et al. (2024) specifically for Telugu. This dataset comprises five hate speech classes: Gender (G), Political (P), Religious (R), Personal Defamation (PD), and Non-hate (NH). The dataset was divided into training and testing sets, with 556 texts and 551 audio samples in the training set, and 50 texts and 50 audio samples in the test set. The paper notes that the dataset is relatively small, which can pose challenges for generalization.

Key Insights and Ideas

  • Multimodal Fusion is Crucial: While unimodal approaches have shown some success, the paper's results suggest that combining text and audio can lead to more robust hate speech detection, especially in nuanced cases.
  • Cross-Modal Attention is Powerful: The proposed cross-modal attention mechanism effectively allows the model to learn the interplay between textual and auditory cues, which is vital for understanding the full context of speech.
  • Transformer Models Excel: The study reinforces the effectiveness of transformer architectures for both text and audio processing in this domain.
  • Low-Resource Languages are a Significant Challenge: The limited size and potential noise in the Telugu dataset highlight the ongoing difficulties in developing high-performing models for low-resource languages.

Key Results

  • Unimodal Text Performance: Among unimodal text models, mBERT achieved the highest F-1 score of 49.68%.
  • Multimodal Performance: The proposed multimodal attention-based approach with Whisper-small and TeluguBERT-3 achieved an F-1 score of 43.68%, securing 3rd place in the shared task.
  • Unimodal Audio Performance: Hubert model achieved the highest F-1 score of 22.94% for unimodal audio.
  • Overall Performance: The authors acknowledge that the overall performance was "unsatisfactory," attributing this partly to the dataset's limited size.

Limitations and Future Work

The authors explicitly mention two key limitations:

  • Dataset Size: The limited size of the dataset restricts the model's ability to generalize and can lead to performance issues.
  • Audio Quality: Noise and recording quality of the audio data can negatively impact performance.

For future work, they suggest:

  • Increasing Dataset Size: Expanding the dataset is crucial for improving generalization.
  • Exploring Other Multimodal Models: Investigating different multimodal architectures could yield better results.
  • Leveraging Larger Language Models (LLMs): The authors believe that exploring various LLMs could further enhance hate speech detection performance in Telugu.

Summary updated

BibTeX
@inproceedings{hossain2025semanticcuetsync,
  author = {Md. Sajjad Hossain and Symom Hossain Shohan and Ashraful Islam Paran and Jawad Hossain and Mohammed Moshiul Hoque},
  title = {{SemanticCuetSync@DravidianLangTech 2025: Multimodal Fusion for Hate Speech Detection - A Transformer Based Approach with Cross-Modal Attention}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure