TL;DR
CamSpokenNLI is a new pilot dataset designed to evaluate Natural Language Inference (NLI) in two widely spoken but under-resourced Cameroonian varieties: Cameroon Pidgin English and Camfranglais. The study reveals that while multilingual models can achieve above-chance reasoning on text, off-the-shelf speech recognition systems like Whisper fail significantly on these varieties, and standard evaluation metrics can easily overstate cross-variety transfer due to "counterpart leakage."
What Problem Does This Paper Address?
Most NLP research focuses on "text-first" languages with standardized scripts. However, many global languages, including Cameroon Pidgin English and Camfranglais, are "oral-first"—primarily spoken rather than written. This creates a massive gap in resources for reasoning tasks like NLI, which determines if one statement (the premise) entails, contradicts, or is neutral toward another (the hypothesis).
Existing resources for related varieties, like Nigerian Pidgin, don't fully capture the unique linguistic blend of Cameroonian varieties, which incorporate significant French and indigenous influences. Furthermore, there are virtually no spoken-premise NLI resources for these languages, making it impossible to build or evaluate voice-based reasoning systems (e.g., for service-exchange platforms).
Methods
The authors took a two-pronged approach to evaluate the current state of AI for these varieties:
- Text-based NLI: They fine-tuned XLM-RoBERTa-base on the reference transcripts of their new dataset. They used a stratified five-fold cross-validation and included "hypothesis-only" and "premise-only" controls to detect if the model was simply exploiting annotation artifacts rather than performing true reasoning.
- Speech Robustness Check: They evaluated zero-shot performance of OpenAI's Whisper (Base, Small, and Large-v3) to see how well general-purpose ASR can recover text from spoken premises in these varieties.
Crucially, they introduced a disjoint-item transfer protocol. Since the dataset contains the same 114 scenarios in both varieties, a naive transfer test would "leak" the scenario from the training set to the test set. Their protocol ensures that if a specific scenario is in the test set for one variety, its counterpart is never seen during training in the other variety.
Datasets
The paper introduces CamSpokenNLI, a pilot dataset consisting of:
- 228 total items: 114 scenario-matched pairs in Cameroon Pidgin English and Camfranglais.
- Rich Annotations: Each item includes premise audio (recorded by a native speaker), premise text, a text hypothesis, an NLI label, and English glosses.
- Balanced Classes: Exactly 38 items each for entailment, contradiction, and neutral labels per variety.
Key Insights and Ideas
- The "Hypothesis-Only" Trap: Models performed nearly as well when only seeing the hypothesis as they did with the full premise-hypothesis pair. This suggests that the wording of the hypotheses (e.g., the presence of negation) might be leaking the labels, a common pitfall in NLI dataset construction that researchers must guard against.
- Counterpart Leakage is Real: When the model was allowed to see the "counterpart" of a test item (the same scenario in the other variety) during training, accuracy jumped by 15–24 percentage points. This "leakage" can lead researchers to believe their models are better at cross-lingual transfer than they actually are.
- ASR is the Bottleneck: Whisper's automatic language identification frequently misidentified Cameroon Pidgin as Yoruba or Swahili. Even with forced language settings, Word Error Rates (WER) remained high, indicating that current ASR must be specifically adapted to these varieties before end-to-end spoken NLI is viable.
Key Results
- NLI Performance: XLM-RoBERTa reached 44.7% accuracy on Pidgin and 46.2% on Camfranglais (chance is 33.3%). While above chance, these results were not statistically distinguishable from hypothesis-only baselines.
- Cross-Variety Transfer: Under strict disjoint-item splits, transfer accuracy was 40.1% (Pidgin to Camfranglais) and 45.6% (Camfranglais to Pidgin).
- ASR Failure: The best zero-shot WER was 66.6% for Pidgin (forced English) and 45.5% for Camfranglais (forced French). Automatic language identification performed even worse.
Limitations and Future Work
The authors are transparent about the "pilot" nature of this work:
- Scale: With only 114 items per variety, the dataset is a "seed resource" rather than a robust benchmark.
- Diversity: All recordings are from a single male speaker from Buea, meaning the data doesn't capture regional dialects, different accents, or varied acoustic conditions.
- Future Directions: The planned version 2.0 aims to include multiple speakers, independent external annotators to mitigate label bias, and unmatched scenarios to prevent models from exploiting parallel alignments. They also intend to move toward end-to-end spoken NLI, performing inference directly from audio without intermediate transcription.
Summary updated