KEC_AI_ZEROWATTS@DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian languages

Kogilavani Shanmugavadivel, Malliga Subramanian, Naveenram C E, Vishal Rs, Srinesh SNAACL 2025

TL;DR

This paper tackles the challenging task of detecting hate speech in Dravidian languages (Tamil, Malayalam, Telugu) by combining text and audio information. They achieved strong results by using a multimodal approach with various machine learning models, demonstrating that integrating different data types significantly improves detection accuracy.

What Problem Does This Paper Address?

Hate speech detection is a critical area in natural language processing, but it becomes significantly more complex when dealing with code-mixed languages, where users blend multiple languages and scripts. Dravidian languages, spoken by millions, present unique challenges due to their linguistic diversity and the unstructured nature of social media content. Existing methods often focus on single modalities (text or audio) or struggle with the nuances of code-mixing. This paper aims to bridge this gap by developing a robust multimodal system for hate speech detection across Tamil, Malayalam, and Telugu, leveraging both textual and acoustic cues.

Methods

The researchers employed a multimodal system architecture with two distinct pipelines: one for text and one for audio.

  • Text Pipeline: Standard text preprocessing techniques were used, including punctuation, URL, and symbol removal. For vectorization, they explored TF-IDF, Count Vectorizer, and Word2Vec. The classification task was handled by traditional machine learning models: Logistic Regression, Ridge Classifier, and Random Forest. These models were chosen for their efficacy and interpretability with high-dimensional text data.
  • Audio Pipeline: Audio data underwent noise reduction, segmentation into smaller chunks, and amplitude normalization. Crucially, prosodic features like pitch, energy, and spectral characteristics were extracted to capture tonal and temporal information relevant to hate speech. Convolutional Neural Networks (CNNs) were used for audio classification.

The final prediction was achieved by combining the outputs from both the text and audio pipelines. To address data imbalance, class balancing techniques were applied, and hyperparameter tuning was performed to optimize model performance.

Datasets

The study utilized a dataset sourced from YouTube videos, containing audio and text samples labeled as either "Non-Hate" or "Hate." The "Hate" category was further subdivided into "Gender," "Political," "Religious," and "Personal Defamation."

  • Text Data: The training dataset comprised 883 records for Malayalam, 1397 for Tamil, and 1953 for Telugu.
  • Audio Data: The training dataset included 883 Malayalam, 509 Tamil, and 551 Telugu recordings.

The paper does not explicitly mention the introduction of a new benchmark dataset but rather the use of an existing one for the shared task.

Key Insights and Ideas

  • Multimodality is Key: The most significant insight is that combining text and audio data dramatically enhances hate speech detection performance compared to using either modality alone. This highlights the complementary nature of linguistic and acoustic cues in identifying harmful content.
  • Language-Specific Performance: Different models excelled in different languages and modalities. For instance, Logistic Regression was top-performing for Tamil text, while Ridge Classifier and CNN showed strengths in specific modalities for other languages. This suggests that language-specific tuning and model selection are crucial.
  • Prosodic Features Matter: The extraction of prosodic features from audio data proved vital, demonstrating that the way something is said (tone, pitch, energy) can be as indicative of hate speech as the words themselves.
  • Traditional ML Still Relevant: Despite the rise of deep learning, traditional machine learning models like Logistic Regression and Random Forest demonstrated strong performance on text data, especially when combined with effective feature engineering.

Key Results

The performance was evaluated using the Macro-F1 score.

  • Tamil: Logistic Regression achieved a Macro-F1 score of 0.97 for text.
  • Malayalam: Random Forest achieved a Macro-F1 score of 0.97 for text.
  • Telugu: Ridge Classifier achieved a Macro-F1 score of 0.89 for text, and CNN achieved 0.87 for audio.

Overall, the multimodal approach led to strong performance across all three languages, with Tamil and Malayalam text models achieving the highest scores. The paper also notes their ranking among 145 teams in the shared task: Tamil (11th), Malayalam (6th), and Telugu (8th).

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: The dataset size is considered minimal, which could restrict the generalizability of the models.
  • Code-Mixed Content: The models may struggle with highly code-mixed content, which is prevalent on social media.
  • Audio Data Preparation: Further optimization of the audio data preparation pipeline could potentially improve the capture of prosodic features.

Future work should focus on:

  • Enhancing Modality Integration: Improving how text and audio information are fused.
  • Dataset Expansion: Growing the dataset to improve model robustness and generalizability.
  • Feature Extraction Optimization: Further refining the extraction of relevant features from both text and audio.
  • Model Calibration: Fine-tuning models for better performance and reliability.

Summary updated

BibTeX
@inproceedings{shanmugavadivel2025kec,
  author = {Kogilavani Shanmugavadivel and Malliga Subramanian and Naveenram C E and Vishal Rs and Srinesh S},
  title = {{KEC\_AI\_ZEROWATTS@DravidianLangTech 2025: Multimodal Hate Speech Detection in Dravidian languages}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure