Most new labeled data for low-resource languages comes from native speakers who correct a model's suggested labels. This setup has a known flaw. When the model is weak, and for languages like Luganda or Wolof it often is, people tend to accept its suggestions, so the model's errors end up in the "gold" data. This note proposes to test a different kind of suggestion interface, borrowed from tutoring research. The assistant phrases its suggestions tentatively ("maybe an organisation?"). It also plants a small, known number of deliberate errors, and it tells annotators when they catch or miss one. The claim is that this keeps the native speaker in charge of the decision. The result should be more correct labels per paid hour than plain suggestions give, and a catch-rate measurement of annotator vigilance comes for free.
The study also targets a second problem. Methods for choosing which sentences to label are almost always evaluated with a simulated annotator that returns the gold label instantly and never defers to the model. We will run the same selection strategies with real native speakers and with the simulated annotator, and check whether the simulation ranks the strategies correctly. One of those strategies is a cheap selection rule taken from evolutionary optimization, opposition-based sampling. It has only been validated on synthetic test functions, so this is also a test of whether that trick survives contact with real language data.
Core Claim
Hedged suggestions with seeded errors and immediate feedback will lower how often annotators accept wrong model suggestions, compared with plain suggestions and at equal annotator time. This will raise downstream named-entity recognition (NER) and sentiment scores. Simulated-annotator evaluation will misjudge the size of the gains that pre-annotation produces, and possibly their order. The new element is treating the suggestion interface as a facilitator in the tutoring sense rather than as a speed-up. We measure its effect on real low-resource annotators, not on a proxy.
Where This Comes From
The facilitation mechanics come from KSC-PaL. It is a peer tutoring agent that uses hedging, feedback requests and deliberate errors to get students to take the lead. It reports that shifts in who leads the conversation explain part of the variance in learning gains ($R^2 = 0.20$). KSC-PaL supplies the method but was only used to teach computer science. Its language understanding also needed a human interpreter behind the scenes. Our setting avoids that problem, because annotators act through a structured labeling interface rather than through free dialogue. We reuse the idea that deliberate errors provoke corrections. The goal changes from learning to accurate labels.
The need for native speakers is motivated by MasakhaNER. It built NER data for 10 African languages through community annotation with very high agreement (Fleiss' kappa 0.96 to 1.0). It found that transfer from a related African language beats transfer from English, and that cross-domain transfer often falls below 40 F1. So native-speaker labels in the target domain are the scarce resource.
Two papers constrain the design. AfroLM removes the human by having the model choose its own training data. Its average NER F1 rises from 68.06 to 80.13, but each retraining round costs about 260 A100-hours. We therefore restrict ourselves to selection rules that do not retrain the model between batches. High-Resource Methodological Bias in Low-Resource Investigations shows that simulating low-resource conditions by down-sampling high-resource corpora can reverse conclusions. In its machine translation experiments, models trained on down-sampled WMT18 did worse on WMT14 than models trained on the real smaller corpus. We apply the same suspicion to the simulated annotator, which is the other standard proxy in active learning. That paper called for better simulation and active learning but designed neither. This study measures how wrong one common simulation is.
The selection rule comes from opposition-based learning. Its key idea is that for any candidate you also evaluate its mirror image across the search box and keep the better of the two. Elite opposition in cuckoo search mirrors only the best candidates. Opposition-based ensemble micro-differential evolution keeps searching broadly with only 6 individuals. A Reinforced Whale Optimization Algorithm keeps whichever of a point and its opposite scores better. All three were validated only on numeric benchmark functions. The closest discrete use is binary opposition-based feature selection for intrusion detection, which works on tabular features. None of them touches text.
Research Program
We will build a web annotation tool for span-level NER and three-way sentiment. It runs in five conditions that differ only in how suggestions appear.
C1 shows no suggestions. C2 shows plain model suggestions. C3 adds seeded errors to plain suggestions but gives no feedback, which is the classic "honeypot" quality check. C4 is the full facilitator: suggestions are phrased tentatively, and when an annotator fixes or misses a seeded error the tool says so right away. C5 is C4 without the tentative phrasing.
Seeded errors appear in a fixed 10% of suggestions and are always plausible. For NER they are type swaps (person to organisation) or boundary shifts. For sentiment, labels sit on an ordinal scale {negative = 0, neutral = 1, positive = 2}, and a seeded error is the mirror label $2 - y$. That maps neutral to itself, so neutral items are never used as seeded errors.
Seeded errors are known, so they are never allowed into the final data. If an annotator accepts one, the item goes back into the queue for another annotator, and that cost is charged to the condition.
The suggestion model is AfroXLMR-large fine-tuned on a small seed set. It stays fixed within each batch.
Selection strategies are compared inside the best interface condition. Random sampling and uncertainty sampling are the standard rules. The opposition rule works as follows. We embed every unlabeled sentence with the frozen encoder and project the embeddings to $d$ principal components. For each coordinate $j$, $a_j$ and $b_j$ are the minimum and maximum of that coordinate over the unlabeled pool. For each of the top-$k$ most uncertain sentences $x$, we form its mirror point
$$x^*_j = a_j + b_j - x_j,$$
and snap $x^*$ to the nearest unlabeled sentence. Of the pair, we keep the one the model is less sure about. This rule costs one nearest-neighbour lookup per selected sentence and no retraining. That matters given AfroLM's compute bill.
We will run on Hausa and Luganda for NER, a large and a small MasakhaNER language, and on Hausa and Xitsonga for AfriSenti. AfriSenti has over 22k Hausa tweets but only about 1.2k Xitsonga tweets.
Annotators re-label sentences drawn from the existing training splits. Their labels can then be scored against high-agreement gold without new adjudication.
Each annotator works through several conditions on disjoint blocks of sentences in counterbalanced order. This reduces the number of annotators we need. We will recruit about six paid native speakers per language, and every condition gets the same total annotator-hours per language.
We also log a direct analogue of KSC-PaL's leadership-shift measure: the fraction of suggested spans or labels the annotator actively changes rather than accepts.
Baselines
The naive reference is C1: native speakers labeling from scratch with no model help. It is slower, but it cannot inherit model errors, so it anchors the accuracy scale.
The strongest practical competitor is C2, plain pre-annotation. It is the default in community annotation because it is fast, and it may already give the most correct labels per hour if the model is good enough.
C3, the silent honeypot, is the strongest simple alternative for the error-seeding idea. It has the same seeded errors as C4 and differs only in feedback and phrasing. C5 separates the effect of tentative phrasing from the effect of feedback.
The model used for suggestions and the downstream model trained on the collected labels are identical across conditions: AfroXLMR-large with the same hyperparameter search budget.
For downstream scores, the published reference points are these. XLM-R base and large models reach roughly 80 to 83 average F1 on MasakhaNER, as reported by MasakhaNER, when trained on full training sets. AfroLM-Large with self-active learning reports 80.13 average NER F1 (AfroLM). On AfriSenti, AfroXLMR-large reaches 71.2 macro F1 (AfriSenti). These are full-data upper references, not budget-matched competitors.
No reported result found on Lacuna for annotator accuracy or labels per hour under model pre-annotation in any African language. The budget-matched numbers will be measured here.
C4 beats C2 if both of the following hold, with confidence intervals from bootstrapping over annotators:
First, C4 accepts fewer genuinely wrong suggestions, meaning the model's own errors rather than the seeded ones.
Second, C4 yields a higher downstream F1 after the same annotator-hours.
For selection, the opposition rule has to beat both uncertainty sampling and a standard diversity rule, greedy k-center, at equal labels.
Why not just…?
Why not just use plain pre-annotation with the best available model? When suggestions are mostly right, accepting them is efficient, and C4 may cost time. The threat is real for Hausa, where suggestions will be decent. It is weaker for Luganda and Xitsonga, where suggestions will be poor. We expect the crossover to depend on suggestion quality. The two NER languages bracket that range, and we will report whichever condition wins at each level rather than a single verdict.
Why not just use hidden honeypot items for quality control? C3 is exactly this. If C3 matches C4 on accepting wrong suggestions, then the tutoring mechanics add nothing beyond measurement. We would then recommend honeypots, since they are simpler. The facilitation hypothesis specifically predicts that feedback and tentative phrasing change behaviour on items that are not seeded, and that difference is what C3 versus C4 tests.
Why not just let the model label its own data, as AfroLM does? Self-active learning produces augmented pretraining text, not native-speaker labels, and it costs about 260 A100-hours per round (AfroLM). We include AfroLM-Large as a reference model. The question here is how to spend human time well, which self-labeling does not answer.
Why not just use uncertainty sampling or k-center instead of opposition-based selection? They may do as well. Mirroring through a bounding box in a projected embedding space has no guarantee of landing anywhere meaningful, and the optimization papers never tested discrete text. We pre-register that if opposition does not beat k-center at equal labels, we report it as a negative result. The interface findings do not depend on it.
Why not just evaluate selection strategies with a simulated gold-label annotator? It is cheap, but it assumes annotators never defer to the model. That is the kind of proxy assumption ter Hoeve et al. showed can flip conclusions. We run every strategy both ways, so the size of the discrepancy is itself a result.
Decisive Checks
The first check is the rate at which annotators accept genuinely wrong suggestions under C2 versus C4, per language. It is scored against gold with kappa of at least 0.96, and a per-language reduction with a confidence interval excluding zero is the minimum evidence for the core claim.
The second check is the learning curve of downstream AfroXLMR-large F1 on the MasakhaNER and AfriSenti test sets against annotator-hours, across C1 to C5.
The third check asks whether the edit fraction predicts label accuracy across annotators. This tests whether KSC-PaL's leadership signal carries over to annotation.
The fourth check is the rank correlation between strategy orderings under real and simulated annotators. A high correlation vindicates simulation, and we will say so. A low one makes real-annotator evaluation the necessary standard for low-resource active learning.
References
- KSC-PaL: A Peer Learning Agent that Encourages Students to take the Initiative
- MasakhaNER: Named Entity Recognition for African Languages
- AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages
- High-Resource Methodological Bias in Low-Resource Investigations
- An Improved Cuckoo Search Algorithm Based on Elite Opposition-based Learning for Indoor Visible Light Positioning
- Opposition-based ensemble micro-differential evolution
- A Reinforced Whale Optimization Algorithm for Solving Mathematical Optimization Problems
- An improved equilibrium optimization algorithm for feature selection problem in network intrusion detection
- AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages