CamSpokenNLI: A Pilot Resource for Cross-Variety Transfer Evaluation in Cameroon Pidgin English and Camfranglais

Tiani Pekins EbikaarXiv 2026 · arXiv

TL;DR

CamSpokenNLI is a new pilot dataset designed to evaluate Natural Language Inference (NLI) in two widely spoken but under-resourced Cameroonian varieties: Cameroon Pidgin English and Camfranglais. The study reveals that while multilingual models can achieve above-chance reasoning on text, off-the-shelf speech recognition systems like Whisper fail significantly on these varieties, and standard evaluation metrics can easily overstate cross-variety transfer due to "counterpart leakage."

What Problem Does This Paper Address?

Most NLP research focuses on "text-first" languages with standardized scripts. However, many global languages, including Cameroon Pidgin English and Camfranglais, are "oral-first"—primarily spoken rather than written. This creates a massive gap in resources for reasoning tasks like NLI, which determines if one statement (the premise) entails, contradicts, or is neutral toward another (the hypothesis).

Existing resources for related varieties, like Nigerian Pidgin, don't fully capture the unique linguistic blend of Cameroonian varieties, which incorporate significant French and indigenous influences. Furthermore, there are virtually no spoken-premise NLI resources for these languages, making it impossible to build or evaluate voice-based reasoning systems (e.g., for service-exchange platforms).

Methods

The authors took a two-pronged approach to evaluate the current state of AI for these varieties:

  1. Text-based NLI: They fine-tuned XLM-RoBERTa-base on the reference transcripts of their new dataset. They used a stratified five-fold cross-validation and included "hypothesis-only" and "premise-only" controls to detect if the model was simply exploiting annotation artifacts rather than performing true reasoning.
  2. Speech Robustness Check: They evaluated zero-shot performance of OpenAI's Whisper (Base, Small, and Large-v3) to see how well general-purpose ASR can recover text from spoken premises in these varieties.

Crucially, they introduced a disjoint-item transfer protocol. Since the dataset contains the same 114 scenarios in both varieties, a naive transfer test would "leak" the scenario from the training set to the test set. Their protocol ensures that if a specific scenario is in the test set for one variety, its counterpart is never seen during training in the other variety.

Figure 1: Overview of the two separate paths in this paper. Top: NLI is run on the reference text of the premises and hypotheses with XLM-RoBERTa-base. Bottom: the premise audio is transcribed by zero-shot Whisper and scored against the premise text. The results are in Sections 4.3 (NLI) and 4.4 (speech check). Neither the audio nor the ASR transcripts are used as NLI input.

Datasets

The paper introduces CamSpokenNLI, a pilot dataset consisting of:

  • 228 total items: 114 scenario-matched pairs in Cameroon Pidgin English and Camfranglais.
  • Rich Annotations: Each item includes premise audio (recorded by a native speaker), premise text, a text hypothesis, an NLI label, and English glosses.
  • Balanced Classes: Exactly 38 items each for entailment, contradiction, and neutral labels per variety.
Figure 2: Label distribution in each variety subset: 114 samples, balanced across entailment, contradiction, and neutral (38 each). Chance level: 33.3%.

Key Insights and Ideas

  • The "Hypothesis-Only" Trap: Models performed nearly as well when only seeing the hypothesis as they did with the full premise-hypothesis pair. This suggests that the wording of the hypotheses (e.g., the presence of negation) might be leaking the labels, a common pitfall in NLI dataset construction that researchers must guard against.
  • Counterpart Leakage is Real: When the model was allowed to see the "counterpart" of a test item (the same scenario in the other variety) during training, accuracy jumped by 15–24 percentage points. This "leakage" can lead researchers to believe their models are better at cross-lingual transfer than they actually are.
  • ASR is the Bottleneck: Whisper's automatic language identification frequently misidentified Cameroon Pidgin as Yoruba or Swahili. Even with forced language settings, Word Error Rates (WER) remained high, indicating that current ASR must be specifically adapted to these varieties before end-to-end spoken NLI is viable.

Key Results

  • NLI Performance: XLM-RoBERTa reached 44.7% accuracy on Pidgin and 46.2% on Camfranglais (chance is 33.3%). While above chance, these results were not statistically distinguishable from hypothesis-only baselines.
  • Cross-Variety Transfer: Under strict disjoint-item splits, transfer accuracy was 40.1% (Pidgin to Camfranglais) and 45.6% (Camfranglais to Pidgin).
  • ASR Failure: The best zero-shot WER was 66.6% for Pidgin (forced English) and 45.5% for Camfranglais (forced French). Automatic language identification performed even worse.
Figure 3: Disjoint-item transfer versus the counterpart- leak audit, which trains on all 114 source items (accu- racy, %). Bars show the mean over seeds 42, 123, and 2024, whiskers the sample SD, and dots the individual seeds. The dashed line is the 33.3% chance level.

Limitations and Future Work

The authors are transparent about the "pilot" nature of this work:

  • Scale: With only 114 items per variety, the dataset is a "seed resource" rather than a robust benchmark.
  • Diversity: All recordings are from a single male speaker from Buea, meaning the data doesn't capture regional dialects, different accents, or varied acoustic conditions.
  • Future Directions: The planned version 2.0 aims to include multiple speakers, independent external annotators to mitigate label bias, and unmatched scenarios to prevent models from exploiting parallel alignments. They also intend to move toward end-to-end spoken NLI, performing inference directly from audio without intermediate transcription.

Summary updated

Abstract

Cameroon Pidgin English (a West African Pidgin English variety) and Camfranglais are widely used in Cameroon, yet to our knowledge, neither has a dedicated spoken Natural Language Inference (NLI) resource. We introduce CamSpokenNLI, a pilot dataset of 114 scenario-matched spoken NLI items in each variety (228 in total), recorded by one native speaker from Buea. Each item pairs premise audio and text with a hypothesis, an NLI label, and English glosses. The author assigned labels, so the dataset is an exploratory resource rather than a mature benchmark. The NLI experiments use the reference text; the audio is evaluated separately. XLM-RoBERTa baselines with five-fold item-level cross-validation reach 44.7% on Pidgin and 46.2% on Camfranglais (chance 33.3%). Still, neither is statistically distinguishable from hypothesis-only input, so hypothesis wording may account for much of the accuracy. Under disjoint-item cross-variety transfer, where a test item's counterpart is never seen in training, accuracy is 40.1% (Pidgin to Camfranglais) and 45.6% (Camfranglais to Pidgin). A counterpart-leak audit raises this to 60-64%, illustrating how counterpart leakage can overstate transfer. Zero-shot Whisper transcription of the recordings performs poorly: the best word error rates are 66.6% for Pidgin and 45.5% for Camfranglais, and with automatic language identification, Whisper Large-v3 labels 93 of 114 Pidgin recordings as Yoruba or Swahili. Together, the results suggest that such resources need disjoint-item evaluation and hypothesis-only controls, and that speech recognition must be adapted to these varieties before spoken NLI can be tested end to end.
BibTeX
@misc{ebika2026camspokennli,
  author = {Tiani Pekins Ebika},
  title = {{CamSpokenNLI: A Pilot Resource for Cross-Variety Transfer Evaluation in Cameroon Pidgin English and Camfranglais}},
  year = {2026},
  eprint = {2611.17573},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17573},
}
Figure