Who Wrote This? Identifying Machine vs Human-Generated Text in Hausa

Babangida Sani, Aakansha Soy, Sukairaj Hafiz Imam, Ahmad Mustapha, Lukman Jibril Aliyu, Idris Abdulmumin, Ibrahim Said Ahmad, Shamsuddeen Hassan Muhammad

TL;DR

This paper introduces the first large-scale machine vs. human text detector for Hausa, a low-resource language spoken by over 100 million people. By fine-tuning Afri-centric transformer models on a newly curated dataset of news articles, the authors achieve up to 99.23% accuracy in distinguishing human-written content from text generated by Gemini-2.0 Flash.

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at generating human-like text, their unregulated use opens the door to plagiarism, fake news, and automated misinformation. To counter this, researchers have developed various AI text detectors, but these tools are overwhelmingly trained on high-resource languages like English or French.

Low-resource languages—even widely spoken ones like Hausa—are often left digitally unprotected. Hausa is a Chadic language spoken predominantly in Northern Nigeria and Niger. From a computational linguistics standpoint, it lacks the massive datasets required to build robust NLP tools. This paper addresses this critical gap by creating the first dedicated framework to detect machine-generated Hausa text in the news domain.

Methods

The authors adopted a model-based classification approach. Instead of using general multilingual models like standard mBERT, they leveraged four Afri-centric pre-trained language models that have been explicitly optimized for African languages:

  • AfriTeVa
  • AfriBERTa
  • AfroXLMR
  • AfroXLMR-76L

To generate the machine-written counterpart of their dataset, the researchers used the Gemini-2.0 Flash model via Google AI Studio. They fed the model real human news headlines and explicitly constrained the generation process so that the output word count would closely match the original human-written articles. This step is crucial because it prevents the downstream classifiers from using text length as a "shortcut" feature for detection.

The models were fine-tuned using standard sequence classification techniques with the AdamW optimizer over 3 epochs.

Training hyperparameters

Datasets

The authors constructed a balanced, bilingual-script-aware dataset consisting of:

  • Human-Generated Texts (HGTs): 2,586 articles scraped from seven local and international online Hausa news outlets (written in the Latin-based Boko script).
  • Machine-Generated Texts (MGTs): 2,586 corresponding articles generated by Gemini-2.0 Flash.

The raw data underwent rigorous preprocessing to remove duplicates, English-language filler text, HTML markup, and API error messages.

Figure 1: Overview of the pipeline’s data collection for human-generated texts.

The final merged dataset contains over 3.3 million words:

Dataset statistics

Key Insights and Ideas

  • The Power of Domain-Specific Pre-training: Multilingual models tailored for African contexts (like AfroXLMR) show remarkable data efficiency. Even with a relatively small fine-tuning dataset (~5,000 total samples), they can achieve near-perfect classification scores.
  • Linguistic Footprints in Low-Resource Settings: The exceptionally high detection accuracy suggests that LLMs leave distinct statistical or stylistic footprints when generating text in low-resource languages. This might be due to the limited Hausa training data available to the LLMs themselves, causing them to rely on more predictable, homogenous sentence structures than human native speakers.

Key Results

All four fine-tuned models demonstrated strong performance, proving highly capable of identifying machine-generated text. AfroXLMR emerged as the top-performing model, achieving an accuracy of 99.23% and an F1 score of 99.21%.

Model performance results

AfriTeVa and AfriBERTa followed closely, while AfroXLMR-76L posted the lowest—yet still highly effective—accuracy at 96.72%.

Limitations and Future Work

The authors acknowledge several limitations in this initial study:

  1. Domain Specificity: The dataset is entirely confined to news articles. The models' performance on social media posts, creative writing, or academic text remains untested.
  2. Single-Generator Bias: Because the machine-generated text was sourced exclusively from Gemini-2.0 Flash, the detector might suffer from over-fitting and fail against texts generated by other LLMs like GPT-4 or Llama.
  3. Hyperparameter Constraints: Due to resource limits, training was restricted to a small batch size (8) and only 3 epochs.

For future work, the team aims to diversify the dataset across multiple domains, test against a wider variety of LLMs, build real-time digital detection frameworks, and explore cross-language transfer learning to protect other underserved African languages.

Summary updated

Abstract

The advancement of large language models (LLMs) has allowed them to be proficient in various tasks, including content generation. However, their unregulated usage can lead to malicious activities such as plagiarism and generating and spreading fake news, especially for low-resource languages. Most existing machine-generated text detectors are trained on high-resource languages like English, French, etc. In this study, we developed the first large-scale detector that can distinguish between human- and machine-generated content in Hausa. We scrapped seven Hausa-language media outlets for the human-generated text and the Gemini-2.0 flash model to automatically generate the corresponding Hausa-language articles based on the human-generated article headlines. We fine-tuned four pre-trained Afri-centric models (AfriTeVa, AfriBERTa, AfroXLMR, and AfroXLMR-76L) on the resulting dataset and assessed their performance using accuracy and F1-score metrics. AfroXLMR achieved the highest performance with an accuracy of 99.23% and an F1 score of 99.21%, demonstrating its effectiveness for Hausa text detection. Our dataset is made publicly available to enable further research.
BibTeX
@inproceedings{sani2025who,
  author = {Babangida Sani and Aakansha Soy and Sukairaj Hafiz Imam and Ahmad Mustapha and Lukman Jibril Aliyu and Idris Abdulmumin and Ibrahim Said Ahmad and Shamsuddeen Hassan Muhammad},
  title = {{Who Wrote This? Identifying Machine vs Human-Generated Text in Hausa}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure