Social-media text in African languages rarely looks like the text models are trained on. The same word appears with and without tone marks, in competing spellings, or swapped mid-sentence for an English or French word. The AfriSenti benchmark authors name missing tone diacritics, code-switching, and dialectal spelling as the main reasons general models fail (AfriSenti). The models that cope best are large Africa-adapted encoders. Deployment, though, usually needs something small. This note proposes a training recipe for that small model. A large adapted "teacher" reads the clean, canonical form of a sentence. A compact "student" reads a deliberately worst-case respelled version of the same sentence and is trained to reproduce the teacher's full output probabilities. The amount of respelling allowed grows separately for each language, and it grows only once the student has caught up at the current level.
In two sentences: we borrow the vision trick of having a robust teacher look at clean inputs while the student looks at attacked ones, but we replace pixel noise with realistic spelling edits searched to be maximally confusing. We then pace how hard those edits get per language, so that languages with 1,200 tweets are not pushed as fast as languages with 22,000. The intended result is a small model that keeps the large model's accuracy on clean text, loses much less under spelling variation, and transfers better to unseen languages.
Core Claim
Our claim is that, for compact low-resource-language classifiers, robustness to orthographic variation is better transferred than trained. Two parts make it transferable. The first is a clean-view teacher signal: soft labels from the teacher on the canonical sentence. The second is a worst-case search over realistic edits, which replaces random noise. A per-language pacing rule, driven by the student's own disagreement with the teacher, adds a further, smaller gain concentrated on the smallest languages. We expect the first two components to carry most of the effect. The curriculum is the most doubtful piece, and the experiments are designed to be able to show that it is unnecessary.
The method draws on three lines of work, each playing a different role. The distillation mechanism comes from vision. In Adversarially Robust Distillation, a student sees attacked images and matches a teacher that sees clean ones; a MobileNetV2 student reaches 50.22% robust accuracy on CIFAR-10, above its own teacher. RSLAD drops hard labels entirely and uses the teacher to guide the attack. We reuse both ideas. Neither paper touches discrete text, where the "perturbation set" must be defined linguistically rather than as a small pixel ball.
The pacing idea comes from Curriculum Adversarial Training. That work grows attack strength gradually and mixes all past strengths into each batch to prevent forgetting, raising CIFAR-10 worst-case accuracy from 46.18% to 69.27%. It uses one global schedule and no teacher. We make the schedule per-language and competence-driven. For the competence signal we follow the student-centered spirit of TAPIR, which measures difficulty as the gap between student and teacher. TAPIR needs a GPT-4 judge to measure that gap; we read it directly off the teacher-student divergence, at no external cost.
Two papers motivate per-language treatment. Long-tailed Adversarial Training with Self-Distillation shows that adversarial training taxes rare classes, and that distilling from a balanced teacher recovers 20.3 points of tail-class robustness on CIFAR-10-LT. The tweet counts across AfriSenti languages are exactly such a long tail. RSLAD also constrains the design: oversized teachers make students underfit. That result is the main argument for a gentle ramp, rather than exposing a small student to the full attack from the start.
The closest cross-lingual work uses adversarial ideas differently. Cross-lingual Multi-Level Adversarial Transfer trains encoders to fool a language discriminator, which is domain-invariance rather than input-robustness. Curriculum Knowledge Distillation for Emoji-supervised Cross-lingual Sentiment Analysis schedules distillation from emoji-heavy to text-heavy teachers, but it has no perturbations and depends on emojis being present. Neither paper asks whether a small model holds up under spelling variation.
Research Program
The first component is an edit model: a set of realistic, label-preserving word-level edits for each language. It covers four kinds of change. The first is stripping or misplacing tone and vowel diacritics, which is relevant to Yoruba and Igbo. The second is common respellings, mined from frequent near-duplicate word forms in the AfriSocial corpus used by AfroXLMR-Social. The third is character-level variants such as dropped vowels and doubled letters. The fourth is code-switch substitution of sentiment-bearing words by their English counterparts, using a small bilingual lexicon. For a sentence $x$, $E_k(x)$ denotes all versions reachable with at most $k$ edits.
Native-speaker spot checks on a sample of edited sentences will confirm that the label is unchanged. Edits that change the label more than a small fraction of the time are discarded. One family of edits, chosen as the respelling family, is held out entirely from training and used only for testing.
The second component is the training objective. The teacher $T$ is AfroXLMR-large, fine-tuned on AfriSenti. The student $S$ is a base-size encoder from the same family. Each language $\ell$ has its own budget $k_\ell$. The student minimizes
$$ \mathcal{L}(x) = \mathrm{KL}\big(T(x)\,\|\,S(x)\big) + \lambda\,\max_{x' \in E_{k_\ell}(x)} \mathrm{KL}\big(T(x)\,\|\,S(x')\big), $$
where $T(x)$ and $S(x)$ are the predicted class distributions and $\lambda$ weights the robustness term. The teacher always sees the clean $x$. The inner maximum is approximated greedily. We rank words by how much the student's gradient-based saliency says they matter, then try edits only on the top few words. This keeps the cost to a handful of extra forward passes per example. Following Curriculum Adversarial Training, each batch mixes examples attacked at every budget from 0 up to $k_\ell$.
The third component is the pacing rule. Every fixed number of steps, we measure on a held-out slice of each language's training data the mean worst-case divergence at the current budget. Language $\ell$'s budget rises by one when that divergence falls below a threshold $\tau$, capped at a maximum $K$. Large languages will therefore reach high budgets quickly, while small ones stay at low budgets until the student has actually fit them.
Baselines
The naive reference point is a base-size student fine-tuned on AfriSenti with ordinary cross-entropy and no teacher. This baseline measures how much any of the machinery buys.
The strongest existing system is the teacher itself, AfroXLMR-large fine-tuned on AfriSenti. The AfriSenti paper reports 71.2 macro F1 for it (AfriSenti). The shared-task leaders report 71.31 weighted F1 on monolingual Task A, 75.06 F1 on multilingual Task B, and 58.15 average F1 on zero-shot Task C for Oromo and Tigrinya (AfriSenti-SemEval). No reported result was found on Lacuna for any model's accuracy under orthographic perturbation on AfriSenti, or for a compact student on AfriSenti, so we will measure both ourselves.
The success criteria differ by comparison. Against the large teacher, beating it on clean text is not expected. Success means a student within about one F1 point of the teacher on clean data that loses less F1 than the teacher under worst-case edits. Against plain distillation, success means a clear robustness gain at equal clean F1.
The most dangerous competitor is noise-augmented training. The student is trained on randomly edited sentences drawn from the same edit model, with two variants: one with hard labels, and one with standard distillation where the teacher sees the edited text. Every system gets the same data, the same student architecture, and the same budget of optimizer steps. Augmentation baselines get extra steps to match the extra forward passes our search uses. Hyperparameters are tuned with an equal number of trials on the same development split.
Three ablations isolate the key ingredients. The first lets the teacher see the edited input instead of the clean one. The second replaces worst-case search with random edits under the same budget. The third replaces per-language competence pacing with a single global linear ramp, and also with no ramp at all.
Why not just…?
Why not just fine-tune the large Africa-adapted model and accept its size? For servers this is fine, and it is our upper reference. However, nothing in the file shows that large adapted models are robust to spelling variation; their benchmark numbers are on the given test text. Our perturbed evaluation will show whether the teacher also degrades. If it does not degrade, the distillation target is sound. If it does, its clean-view soft labels are still the right supervision for the student.
Why not just augment the training data with random spelling noise? This is the real threat, and we expect it to recover a large share of the gain. The evidence for going further is indirect. In vision, random noise during training is a much weaker defense than attacks searched to be worst-case. In ARD, plain distillation gives 1.30% robust accuracy on CIFAR-100 against 27.64% for adversarial distillation. Discrete spelling edits may behave differently, though. If random augmentation plus distillation matches our method on worst-case and held-out edits, we will report that the search is unnecessary and recommend the cheaper recipe.
Why not just apply adversarial training directly to the small model with hard labels? Prior work suggests that a capacity-limited model pays a large clean-accuracy price under direct adversarial training (ARD, RSLAD). The same work suggests the price falls hardest on rare groups (Long-tailed AT with Self-Distillation). The hard-label adversarial baseline tests this directly, through clean F1 on the smallest languages.
Why not just use one fixed schedule, or none? Curriculum gains shrink as data grows (Dynamic Curriculum Learning for Low-Resource NMT found negligible gains at 4.5M sentence pairs). Batch mixing alone might therefore suffice. The pacing ablation will settle this, and we predict any benefit appears only on the languages with the fewest tweets.
Proposed Experiments
The primary setting is AfriSenti across all 14 languages. We report four metrics: clean macro F1, F1 under random edits, F1 under worst-case edits at budgets of one through five, and F1 under the held-out respelling family. The worst-case search used at test time attacks each system's own predictions, so no system is attacked with its own training search. Zero-shot transfer is measured on the Task C languages Oromo and Tigrinya.
A secondary check uses the NGLUEni text-classification tasks (NGLUEni), with zero-shot transfer to isiNdebele and Siswati. There, the teacher is the Nguni-adapted model and the edit model covers Nguni orthographic variants.
The decisive check is the comparison against random-noise augmentation with distillation, specifically on the held-out respelling family and the zero-shot languages. A real advantage there means the method has learned robustness rather than memorizing the training edits. Absent that advantage, the core claim fails. A second check plots robustness gain against language size: per-language pacing should matter most for Xitsonga-scale languages and not at all for Hausa. A third check measures compute overhead in wall-clock time against the baselines, since an ~30% re-ranking cost was the main practical objection to dynamic curricula in prior low-resource work (Dynamic Curriculum Learning for Low-Resource NMT).
References
- AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages
- Adversarially Robust Distillation
- Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
- Curriculum Adversarial Training
- Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning
- Long-tailed Adversarial Training with Self-Distillation
- Cross-lingual Multi-Level Adversarial Transfer to Enhance Low-Resource Name Tagging
- Curriculum Knowledge Distillation for Emoji-supervised Cross-lingual Sentiment Analysis
- AfroXLMR-Social: Adapting Pre-trained Language Models for African Languages Social Media Text
- SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)
- Dynamic Curriculum Learning for Low-Resource Neural Machine Translation
- NGLUEni: Benchmarking and Adapting Pretrained Language Models for Nguni Languages