TL;DR
This paper presents NAYEL, a simple yet effective character n-gram TF-IDF and ensemble machine learning model for detecting fake news in Dravidian languages. Despite its simplicity, the model achieved a strong F1-score of 87.5%, securing 5th rank in the shared task, highlighting the potential of basic approaches for under-resourced languages.
What Problem Does This Paper Address?
The proliferation of social media has amplified the spread of fake news, posing significant challenges to public trust and governance. While much research has focused on high-resource languages like English, the unique linguistic and cultural characteristics of regional languages, particularly Dravidian languages (Tamil, Telugu, Kannada, Malayalam), have been largely overlooked. These languages present specific NLP challenges due to their diverse dialects, scripts, and cultural nuances, making fake news detection a complex task. This paper addresses this gap by proposing a model specifically for fake news detection in Dravidian languages.
Methods
The NAYEL model employs a straightforward yet effective approach. It utilizes character n-gram TF-IDF for feature extraction, specifically considering 3-gram, 4-gram, 5-gram, and 6-gram combinations. This method treats each n-gram as a token, capturing sub-word patterns that can be beneficial for languages with rich morphology.
The extracted features are then fed into an ensemble of classical machine learning classifiers. The base classifiers include Support Vector Machines (SVM), Naive Bayes (NB), and a linear classifier with Stochastic Gradient Descent (SGD). The outputs of these individual classifiers are combined using a majority voting mechanism to produce the final prediction. This ensemble approach aims to leverage the strengths of each individual model.
Datasets
The dataset used in this task was collected from various social media platforms, including X (formerly Twitter) and Facebook. The paper refers to a detailed description and analysis of the dataset provided in a separate publication (Subramanian et al., 2025). The dataset is divided into two subtasks:
- Task A: Classifying social media text into "Original" or "Fake."
- Task B: Detecting fake news from Malayalam News into five fake categories plus "Original."
The authors participated in Task A and utilized a development set for hyperparameter tuning. Table 1 provides statistics for both tasks, showing the distribution of "Fake" and "Original" classes for Task A, and "Half True," "False," "Partly False," and "Mostly False" for Task B, across training, testing, and development sets.
Key Insights and Ideas
- Simplicity Can Be Powerful: The paper demonstrates that a relatively simple model architecture (character n-grams + ensemble ML) can achieve competitive results, even outperforming more complex structures in some cases. This is particularly relevant for low-resource languages where complex models might be harder to train and deploy.
- Character N-grams for Dravidian Languages: The choice of character n-grams proved effective, likely due to its ability to capture morphological variations and sub-word patterns common in Dravidian languages, which might be missed by word-level features.
- Ensemble Learning's Value: The majority voting ensemble successfully combined the strengths of individual classifiers, leading to improved overall performance, especially in terms of the F1-score.
Key Results
The proposed NAYEL model achieved a F1-score of 87.5% on the test set for Task A, securing the 5th rank in the Fake News Detection in Dravidian Languages shared task. On the development set, the individual classifiers showed varying strengths: SGD achieved the minimum results, SVM excelled in precision, NB had the highest recall, and both SVM and the voting classifier achieved the highest F1-score of 0.88. The majority voting model ultimately outperformed all baseline models on the test set.
Limitations and Future Work
The authors acknowledge several limitations:
- Semantic Feature Limitation: The character n-gram TF-IDF approach is primarily lexicographical and does not capture semantic meaning, which can be crucial for nuanced fake news detection.
- Dataset Quality: The quality of the dataset is highlighted as a significant factor in developing robust systems.
Future work suggested includes:
- Improved Pre-processing: Implementing more advanced text cleaning techniques.
- Exploring More ML Algorithms: Investigating a wider range of classical machine learning algorithms.
- Leveraging LLMs: Testing the effectiveness of Large Language Models (LLMs) for this task, acknowledging that limited computational resources currently favor classical ML approaches.
Summary updated