Which Parts of Pragmatic Skill Survive Translation into Low-Resource Languages?

Language models for low-resource languages are mostly built from "textbook" text: machine translations of English, government notices, news, and religious material. This resembles the grammar-drill classroom that language teachers criticize, where students score well on structure but cannot use the language with people. Such data teaches a model to produce well-formed isiZulu or isiXhosa. It says little about what speakers mean when they answer a request indirectly, how respect is marked toward elders, or when a literal reply is rude. We propose to measure this gap directly and then ask a practical question that current work has not addressed. When native-speaker annotation is the scarcest resource, which pragmatic abilities can be imported cheaply from English or machine translation, and which must be written by native speakers?

Our hypothesis is that pragmatic competence splits into two parts. The first is a largely language-general inference skill, such as reading "I have an early meeting" as a "no" to a late-night invitation. The second is a set of local conventions: respect registers, indirect refusal formulas, and culturally specific readings of the same words. We expect the first to transfer from English instruction data and the second not to. If this holds, a few hundred native-authored examples aimed only at local conventions should outperform thousands of translated examples. That gives low-resource communities a concrete rule for where to spend annotation effort.

Core Claim

The claim has two parts, and each can be falsified on its own.

The first is diagnostic. Continued pre-training, meaning further training of an existing model on text in the target languages, raises structural and general benchmark scores more than it raises pragmatic scores when the training mixture is dominated by translated text. The result is a text-only version of the "fluent fool" that MELLA documented for vision-language models. MELLA showed that image-text models trained on translated captions became fluent but culturally illiterate, and that locally harvested alt-text restored cultural recognition. It did not study dialogue, implied meaning, or text-only models. AfriqueLLM found that synthetic translations are necessary for reasoning during continued pre-training on African languages, but it never measured pragmatics. The two findings together suggest that the ingredient which rescues reasoning could also flatten meaning into literal form.

The second is about how to allocate scarce data. The Goldilocks of Pragmatic Understanding showed that implicature resolution does not emerge from base training or from benchmark-style instruction tuning. It appears only with example-level instruction tuning, where every training example carries its own natural, context-specific instruction, combined with step-by-step reasoning. We predict this recipe transfers across languages for the language-general inference part but not for local conventions. We therefore expect a small native-authored convention set to beat a large translated set on convention items, while giving roughly equal results on language-general items.

Research Program

We work in isiZulu and isiXhosa, with Siswati as a lower-resource probe. Three things make these languages a good fit. NGLUEni provides structural and generation benchmarks for them. NGLUEni also shows that adaptation within the language family transfers well to Siswati. And AfriqueLLM supplies models before and after continued pre-training, so we can compare them directly.

The first artifact is a paired pragmatics test set, built with native speakers, with two matched halves per language.

The "translatable" half is built from the English binary implicature items of George and Mamidi used in Goldilocks. In each item, a question receives an indirect answer that means yes or no. Native speakers post-edit the translations and discard items whose English framing does not carry over.

The "convention" half is authored from scratch by native speakers in the same yes/no format and in a short free-response format. It covers exchanges whose correct reading depends on local practice, such as indirect refusals, respect-register choices across age and kinship, and greetings whose omission signals something. Examples in the Goldilocks style keep the format identical across halves, so any difference in scores reflects content rather than task format.

We target about 600 items per half per language, each labeled by at least three speakers. We report agreement and treat items below a set agreement threshold as ambiguous. For the collection process, we borrow the idea of a learning-annotation loop from EnetCollect. EnetCollect turned learner exercises into structural annotations for Ladin and South Tyrolean German. We apply the same idea to pragmatic judgments. Heritage learners judge whether a reply is appropriate, and fluent speakers adjudicate. This extends EnetCollect from part-of-speech and lexicon annotation to appropriateness judgments. The test sets themselves rest on paid native annotators.

The second artifact is a set of training pools, each in its own role:

  • English pool. Example-level implicature instructions in English, with reasoning chains.
  • Translated pool. The same instructions machine-translated into the targets.
  • Native pool. Native-authored convention items, disjoint from the test set, collected at budgets of 100, 300, and 1,000.
  • Control pool. A native-authored set of the same sizes that covers only language-general implicatures. This control separates "written by a native speaker" from "about a local convention."

The diagnostic study evaluates several models on both halves and on structural and general benchmarks: Qwen 3 8B base, AfriqueQwen-8B after continued pre-training, and the larger AfriqueQwen-14B. The benchmarks are AfroBench-Lite and NGLUEni headline and data-to-text generation. For any change in pragmatic score, we express it as a fraction of the structural or general gain the same model achieves. This ratio is the "fluent fool" measure.

The same evaluation also informs the unresolved tension between The Pragmatic Mind of Machines and Goldilocks. The first paper argues that pragmatic skill emerges with pre-training scale. The second finds that base models sit near chance on implicatures. In low-resource languages, the pre-training volume the first paper relies on will never exist, so emergence from scale may not occur there at all.

Baselines

The naive reference is each model prompted zero-shot on both halves with no pragmatic tuning. Goldilocks reports base models around 60% on English binary implicatures, against 50% chance. We expect low-resource performance to be at or below that level. No reported result was found on Lacuna for implicature resolution in any Nguni language.

The strongest existing method is the Goldilocks recipe: example-level instruction tuning plus five-shot step-by-step prompting. Goldilocks reports GPT-4 with five-shot step-by-step prompting at 86.5% accuracy on the English implicature set, against a human baseline of 86.2%. We run two versions. The first is a frontier model prompted in the target language with five translated step-by-step examples. The second is AfriqueQwen-8B fine-tuned on the English pool, which is the direct test of cross-lingual transfer.

The translated baseline fine-tunes on the translated pool at two sizes: the same size as the largest native budget, and ten times larger.

The benchmark-level baseline is one template applied to a whole dataset, Flan-style. Goldilocks found this setup near 60% in English, so it serves as a check that our format behaves as expected.

All fine-tuned conditions share the same base checkpoint, number of optimizer steps, learning-rate sweep, and number of seeds. Test items never appear in any pool.

The key ablation swaps the native convention pool for the native control pool at equal size. This isolates convention content from native authorship.

"Beating" the strongest baseline has a specific meaning. On the convention half, English-plus-native-300 must exceed both English-only and English-plus-translated-3,000 by a margin larger than the seed-to-seed spread and the annotator-disagreement band. On the translatable half, the difference should be small. The convention-pool ablation must account for most of the convention-half gain.

Why not just…?

Why not just translate English pragmatic datasets into the target languages? This is the real threat, and the design tests it head-on rather than assuming it fails. MELLA's evidence argues against it. Translated captions alone left Czech cultural keyword accuracy at 1.10%, and adding native alt-text raised it to 15.56%. But that result comes from entity recognition in images, not from dialogue. The ten-times-larger translated pool is the fair test. If it matches the native pool on the convention half, our claim fails, and the useful conclusion becomes "translate, and don't pay for native authoring." We would report it that way.

Why not just prompt a frontier multilingual model with step-by-step examples? Goldilocks shows this reaches human level in English. SocialCC found that strong models know cultural facts yet fail to act on them in interaction, and that they carry a secular, Western tilt. The AfriSenti zero-shot track peaked at 58.15 average F1, compared with 75.06 with multilingual training, according to AfriSenti-SemEval. These results suggest weakness on local conventions, but none of them measures implicature in these languages. The frontier-prompting baseline measures this directly. It also serves as a practical ceiling that an open 8B model would need to approach to justify local tuning.

Why not just tune in English and rely on cross-lingual transfer? Our own hypothesis predicts this will work for the translatable half, so it is not a rival there. It is the condition that separates what transfers from what does not. If English-only tuning also closes the convention gap, then local conventions are less local than we assume, and the native data budget would be better spent on structural resources.

Why not just add native monolingual text to continued pre-training instead? AfriqueLLM reports that adding monolingual data alone during continued pre-training degraded reasoning through forgetting. Goldilocks shows that raw text exposure does not produce implicature resolution. We therefore do not expect unlabeled text to teach conventions efficiently, and we do not spend compute on this route beyond the diagnostic comparison of models before and after continued pre-training.

Proposed Experiments and Decisive Checks

The first check is the diagnostic ratio. We compare AfriqueQwen-8B with Qwen 3 8B base on each benchmark. The prediction is that AfroBench-Lite and NGLUEni generation scores rise substantially, while the convention half rises by a much smaller proportion. AfriqueLLM reports a 76.5% relative improvement for AfriqueQwen-8B over its base on AfroBench-Lite. If the convention half improves by a similar proportion, the "fluent fool" framing is wrong for text models.

The second check is the transfer split. Tuning on the English pool should raise the translatable half substantially and the convention half only slightly. We quantify this as the difference between the two halves' gains, with bootstrap confidence intervals over items.

The third check is the budget curve. We plot convention-half accuracy against native budgets of 100, 300, and 1,000, alongside the translated pool at one and ten times the size. The claim holds if native-300 is above translated-3,000, and if the convention-pool ablation, not the control pool, produces the gain.

The fourth check guards against the loss of general ability documented in AfriqueLLM and in Efficient Low-Resource Language Models Using Tokenizer Transfer. We re-run AfroBench-Lite and NGLUEni after each tuning condition. A pragmatic gain that costs general ability does not count as a win.

Siswati receives no native training data, only test items. Evaluating on it shows whether native conventions taught in isiZulu and isiXhosa carry over within the language family, as NGLUEni's structural transfer results would suggest.

Across all four checks, the outcome that matters is a clear statement of which pragmatic skills a low-resource community can obtain through translation, and which only its own speakers can supply.