TL;DR
This paper introduces the first large-scale machine vs. human text detector for Hausa, a low-resource language spoken by over 100 million people. By fine-tuning Afri-centric transformer models on a newly curated dataset of news articles, the authors achieve up to 99.23% accuracy in distinguishing human-written content from text generated by Gemini-2.0 Flash.
What Problem Does This Paper Address?
While Large Language Models (LLMs) excel at generating human-like text, their unregulated use opens the door to plagiarism, fake news, and automated misinformation. To counter this, researchers have developed various AI text detectors, but these tools are overwhelmingly trained on high-resource languages like English or French.
Low-resource languages—even widely spoken ones like Hausa—are often left digitally unprotected. Hausa is a Chadic language spoken predominantly in Northern Nigeria and Niger. From a computational linguistics standpoint, it lacks the massive datasets required to build robust NLP tools. This paper addresses this critical gap by creating the first dedicated framework to detect machine-generated Hausa text in the news domain.
Methods
The authors adopted a model-based classification approach. Instead of using general multilingual models like standard mBERT, they leveraged four Afri-centric pre-trained language models that have been explicitly optimized for African languages:
- AfriTeVa
- AfriBERTa
- AfroXLMR
- AfroXLMR-76L
To generate the machine-written counterpart of their dataset, the researchers used the Gemini-2.0 Flash model via Google AI Studio. They fed the model real human news headlines and explicitly constrained the generation process so that the output word count would closely match the original human-written articles. This step is crucial because it prevents the downstream classifiers from using text length as a "shortcut" feature for detection.
The models were fine-tuned using standard sequence classification techniques with the AdamW optimizer over 3 epochs.
Datasets
The authors constructed a balanced, bilingual-script-aware dataset consisting of:
- Human-Generated Texts (HGTs): 2,586 articles scraped from seven local and international online Hausa news outlets (written in the Latin-based Boko script).
- Machine-Generated Texts (MGTs): 2,586 corresponding articles generated by Gemini-2.0 Flash.
The raw data underwent rigorous preprocessing to remove duplicates, English-language filler text, HTML markup, and API error messages.
The final merged dataset contains over 3.3 million words:
Key Insights and Ideas
- The Power of Domain-Specific Pre-training: Multilingual models tailored for African contexts (like AfroXLMR) show remarkable data efficiency. Even with a relatively small fine-tuning dataset (~5,000 total samples), they can achieve near-perfect classification scores.
- Linguistic Footprints in Low-Resource Settings: The exceptionally high detection accuracy suggests that LLMs leave distinct statistical or stylistic footprints when generating text in low-resource languages. This might be due to the limited Hausa training data available to the LLMs themselves, causing them to rely on more predictable, homogenous sentence structures than human native speakers.
Key Results
All four fine-tuned models demonstrated strong performance, proving highly capable of identifying machine-generated text. AfroXLMR emerged as the top-performing model, achieving an accuracy of 99.23% and an F1 score of 99.21%.
AfriTeVa and AfriBERTa followed closely, while AfroXLMR-76L posted the lowest—yet still highly effective—accuracy at 96.72%.
Limitations and Future Work
The authors acknowledge several limitations in this initial study:
- Domain Specificity: The dataset is entirely confined to news articles. The models' performance on social media posts, creative writing, or academic text remains untested.
- Single-Generator Bias: Because the machine-generated text was sourced exclusively from Gemini-2.0 Flash, the detector might suffer from over-fitting and fail against texts generated by other LLMs like GPT-4 or Llama.
- Hyperparameter Constraints: Due to resource limits, training was restricted to a small batch size (8) and only 3 epochs.
For future work, the team aims to diversify the dataset across multiple domains, test against a wider variety of LLMs, build real-time digital detection frameworks, and explore cross-language transfer learning to protect other underserved African languages.
Summary updated