Abstract: The rise of social media has amplified the spread of fake news, now further complicated by large language models (LLMs) like ChatGPT, which ease the generation of highly convincing, error-free misinformation, making it increasingly challenging for the public to discern truth from falsehood. Traditional fake news detection methods relying on linguistic cues also becomes less effective. Moreover, current detectors primarily focus on binary classification and English texts, often overlooking the distinction between machine-generated true vs. fake news and the detection in low-resource languages. To this end, we updated detection schema to include machine-generated news with focus on the Urdu language. We further propose a hierarchical detection strategy to improve the accuracy and robustness. Experiments show its effectiveness across four datasets in various settings.
Introduction
Fake news detection has traditionally been treated as a binary classification problem (True vs. Fake) focusing primarily on English. However, the rise of Large Language Models (LLMs) like GPT-4o has changed the landscape. Propagandists can now generate high-quality, "journalistic-style" misinformation that lacks the typical linguistic errors human-authored fake news might have.
In low-resource languages like Urdu, the challenge is even steeper. Existing Urdu datasets only contain human-written text, leaving a massive gap in our ability to detect machine-authored misinformation. Furthermore, standard "flat" classifiers often struggle to distinguish between machine-generated true news and machine-generated fake news, frequently misidentifying one for the other.
The Hierarchical Detection Strategy
To address these challenges, we moved away from a simple four-class "flat" model (Human True, Human Fake, Machine True, Machine Fake) in favor of a Hierarchical Detection Strategy. This approach allows each model to focus on specific features—linguistic patterns for machine detection and factual/contextual patterns for veracity—rather than trying to learn both simultaneously.
- The Architecture: The problem is split into two specialized sub-tasks:
- Sub-task A: Machine-Generated Text (MGT) Detection (Human vs. Machine)
- Sub-task B: Fake News Detection (True vs. Fake)
- The Backbone: Both sub-tasks utilize a fine-tuned xlm-ROBERTa-base model, chosen for its strong multilingual capabilities.
- Inference: At the prediction stage, the outputs of these two models are concatenated to produce the final four-way label (Human_True, Human_Fake, Machine_True, Machine_Fake).
Dataset Construction
A new benchmark was created by augmenting four existing Urdu news datasets: Ax-to-Grind Urdu (10,083 samples), UFN2023 (4,097 samples), UFN Augmented Corpus (2,000 samples), and Bend the Truth (1,300 samples). To create the machine-authored components, we used GPT-4o with five distinct prompts to paraphrase the original articles while maintaining their original stance. This effectively doubled the size of the datasets and created the first Urdu corpus containing "Machine True" and "Machine Fake" labels.
Experimental Results
The hierarchical model achieved an overall accuracy of 87% when trained on all datasets combined, compared to 81% for the flat xlm-ROBERTa baseline. On the "Machine True" (MT) class—the hardest to predict—the hierarchical model often saw double-digit improvements in F1-score over the baseline.
Key insights from the experiments include:
- The "Machine-Generated" Blind Spot: Baseline models performed significantly worse on machine-generated classes than human ones. Machines write "too well," lacking the emotional or linguistic "tells" often found in human-written fake news.
- The Length Trap: In cross-domain testing, models often failed because they inadvertently learned to use text length as a proxy for veracity or authorship.
- Hierarchical Superiority: By separating "who wrote it" from "is it true," the hierarchical model narrowed the performance gap between human and machine classes. It achieved nearly identical F1 scores for both, proving that MGT detection is a distinct signal that shouldn't be buried in a multi-class label.
Conclusion and Future Work
While the hierarchical approach showed better generalization than the baseline, cross-domain performance remains a challenge, especially between short headlines and long articles. Future work will focus on increasing dataset diversity beyond GPT-4o, mitigating length bias, and integrating explainable AI (XAI) to better understand the model's decision-making process in the Urdu language.
Summary updated