Small abuse classifiers for low-resource languages are increasingly trained on labels produced by a large language model, the "teacher," because native-speaker annotation is scarce. Those labels carry the teacher's cultural blind spots. One failure is over-flagging: the teacher marks local banter and colloquial insults as hate. The other is under-flagging: it misses ordinary words that a community uses as slurs. Many of these judgments also depend on the surrounding conversation. Our proposal is to label every message several times, each time showing the teacher a different amount of conversational history: none, the sender's own prior turns, the preceding few turns, and the whole thread. The messages whose label flips across these views are a cheap, label-free map of where the teacher's synthetic labels cannot be trusted. That map can steer a small human-annotation budget and shape what the small "student" model is trained on.
This matters because two well-documented effects meet on exactly these messages. Conversational context changes what a message means, so the right amount of history is task-specific. Culturally coded abuse is often invisible without the setting in which it is said. If inherited bias concentrates on context-dependent messages, then deciding how much history to show the teacher is not only an efficiency question. It is the main lever on what bias the student inherits.
Core Claim
The claim has three parts. First, in conversational abuse detection for dialectal and low-resource languages, the errors a distilled student inherits from an LLM teacher are concentrated on messages whose teacher label is unstable across context windows. Second, this instability predicts teacher error better than the teacher's confidence, at matched inference cost. Third, a student trained with context-instability-aware supervision closes more of the gap to human-label training than global fixes that treat inherited bias as a uniform offset. Those global fixes are threshold recalibration and demographic stratified sampling.
We also expect the size of this effect to depend on how much a language community leaves meaning implicit. We test this with an existing pronoun-drop measure of high- versus low-context communication.
Grounding in Prior Work
Several papers motivate the proposal by showing that context decides meaning in abuse detection. Hate Speech and Counter Speech Detection: Conversational Context Does Matter built an English Reddit set of parent–target comment pairs. In that set, showing annotators the parent comment changed their label 38.3% of the time, and adding the parent raised a RoBERTa classifier's weighted F1 from 0.58 to 0.64. The study used only a two-turn window and only English.
Context-Aware Toxicity Detection in Multiplayer Games found that 67% (DOTA 2) and 89% (Call of Duty) of toxic messages are recognisable only with preceding turns. It also reported a "More Context Paradox": the whole match chat did worse than the sender's own history alone. Both papers use human labels in English, and neither asks what happens when the labels come from a model.
A second group of papers motivates the cultural side. Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish shows an LLM teacher (Aya) at 93.0% recall but 11.9% precision on Argentine news replies, over-flagging regional slang. It explicitly leaves open whether distillation into a small encoder such as BETO transfers that false-positive bias. AfriHate and Muti et al.'s work on Italian misogyny (Muti et al.) show that innocuous words become slurs only in particular social settings. "Aboki" means "friend" in Hausa but can be an ethnic slur; balena means "whale" but is used as a misogynist insult. These are precisely the cases where the label should depend on context.
A third group of papers constrains the design. SEAHateCheck found that fine-tuning on LLM-generated "silver" data caused regressions when that data diverged from dialectal nuance. Its tests are single sentences, so it cannot say whether the regressions lie in context-dependent cases. Fairness Feedback Loops shows that training on model-generated data widens group error gaps, reporting a 15% accuracy drop and erasure of a minoritized group across generations. Its fix, stratified sampling (STAR), is applied on image and tabular data and trades accuracy for parity. We reuse STAR as a baseline rather than as our method, and we test whether it addresses context-dependent errors at all.
From the context-windowing literature, CORAL and Towards Adaptive Context Management show inverted-U behaviour in English retrieval and question answering: performance rises with history, then falls, with degradation after roughly 4–6 turns in CORAL. They treat window choice as an accuracy knob for one model. We treat disagreement across windows as a signal about label reliability. To our knowledge, nobody has done that.
Finally, Cross-Cultural Similarity Features supplies the method for the cultural covariate. Its Language Context-level Ratio, a part-of-speech measure of how often a language drops pronouns, stands in for the high- versus low-context communication divide.
Research Program
The pipeline has three stages.
Stage 1: Multi-window teacher labelling. For each message in a threaded corpus, we query the teacher under four context views: target only, sender history, the preceding $k$ turns (with $k$ swept from 1 to 8), and the full thread. We record the label, the verbalised confidence, and any chain-of-thought rationale under each view.
Stage 2: Defining context-instability. We define a message's context-instability $u$ as the entropy of the teacher's labels across the window views. A message with $u = 0$ gets the same label however much history the teacher sees. We separately record the direction of each flip, for example "hateful in isolation, benign with context." Direction matters because over-flagging of banter and missed coded slurs should flip in opposite directions.
Stage 3: Training the student. Students are compact encoders (BETO and RoBERTuito for Spanish, AfroXLMR for African languages) that receive the window the deployment can afford. We compare three ways of training them on a fixed human-annotation budget $B$:
- Disagreement-routed annotation: spend $B$ on the highest-$u$ messages and use teacher labels elsewhere.
- Soft multi-window targets: train the student on the teacher's label distribution across windows rather than a single label.
- Combined: both of the above together.
To separate inheritance from ordinary error, we measure how far student errors land on teacher errors beyond what independent mistakes would produce:
$$I = \frac{P(\text{student wrong} \mid \text{teacher wrong})}{P(\text{student wrong} \mid \text{teacher right})}$$
Here both probabilities are estimated on human-labelled test items. A value of $I = 1$ means the student's mistakes are independent of the teacher's, and larger values mean stronger inheritance. We report $I$ separately for high- and low-instability strata.
Data. The data component is real work, and we say so. The Rioplatense corpus consists of replies to news posts. If its thread structure can be recovered, it gives native multi-turn context for a dialect with documented teacher failure. The Reddit parent–target set is a low-context English anchor with an existing human context-flip rate. We will also annotate a new threaded evaluation set of a few thousand messages in Egyptian Arabic, Hausa, and Tagalog. These languages are chosen to span the Language Context-level Ratio range, and the set will be annotated by native speakers both with and without thread context, following the Reddit study's protocol. This gives a human context-flip rate per language, which we can compare against the teacher's.
Baselines
The naive reference point is a student trained on teacher labels produced from the target message alone. This is how silver data is usually built, as in SEAHateCheck. A second reference is the teacher itself, applied zero-shot to the test sets. On Rioplatense Spanish the reported figure for the teacher is 93.0% recall and 11.9% precision for Aya (Pérez et al.).
The strongest human-supervised competitor is the Parent+Target RoBERTa recipe from the Reddit study, which reports weighted F1 of 0.64 on English Reddit. We port this recipe to each student encoder and train it on the same human budget $B$ used to route annotations, so the comparison asks whether teacher labels plus routing beat human labels alone. For the game-chat context paradox, the file reports no balanced-accuracy number. For distilled students on multi-turn non-English abuse data, no reported result was found on Lacuna.
The strongest bias-mitigation competitors are three methods that need no knowledge of context:
- Global threshold recalibration: fit the decision threshold on $B$ gold items.
- Teacher-confidence filtering: drop low-confidence teacher labels, or route them to annotators.
- CLA-STAR stratified sampling: from Fairness Feedback Loops, stratified by annotated target group.
All competitors use the same teacher, the same student architecture, the same gold budget, and the same hyperparameter-search budget, selected on the same development split.
Three ablations isolate the key ingredient:
- Replace context-instability with sampling disagreement: re-query the teacher several times with the window held fixed, using as many queries as the multi-window method. This tests whether context variation matters or just any repeated querying.
- Route the budget at random.
- Use multi-window labels while discarding flip direction.
What counts as beating the baselines. The method wins if, at equal budget, it achieves higher macro-F1 and a lower inheritance ratio $I$ on high-instability items than every baseline, with confidence intervals from multiple seeds. It must also show higher precision on in-group banter without losing recall on context-coded slurs.
Why not just…?
Why not just give the teacher the full thread and use its labels? This may be the right answer for some languages, and the window sweep tests it directly. The game-chat paradox and CORAL's degradation after a handful of turns suggest that more history injects noise. But those results come from fine-tuned or retrieval systems, not LLM labellers, so we do not assume they transfer. If full-thread labels match the multi-window method on $I$ and macro-F1, we will report that window choice suffices and that instability adds nothing.
Why not just recalibrate the decision threshold on a small gold set? This is the most serious threat. The Aya precision collapse looks like a threshold problem, and a recalibrated threshold may recover much of the precision. Our claim survives only if inherited errors are not a uniform shift. Banter should flip toward "benign" with context, while coded slurs flip toward "hateful," so one threshold cannot fix both. We will report per-direction and per-target-group error after recalibration. If recalibration matches our method on both, the proposal's central claim fails.
Why not just use teacher confidence or self-consistency to find bad labels? Confidence and sampling disagreement are cheap, well-known noise detectors. The ablation runs them with the same number of teacher queries as our method. We compare how well each predicts teacher error (AUROC against human labels), separately on context-dependent and context-independent items. Our hypothesis is that confidence misses confidently wrong judgments made in isolation, such as reading trola or "Aboki" literally. If confidence predicts teacher error as well as context-instability does, we fold it into the method instead of claiming separation.
Why not just annotate everything by hand? At the budgets that motivate distillation, full annotation is not available, which is why silver data exists. We nevertheless plot the full budget curve up to complete human labelling. That shows where, if anywhere, routing stops mattering.
Decisive Checks
The study has two decisive checks.
Check 1: Is the instability signal real? Teacher context-instability should correlate with the human context-flip rate measured in our with/without-context annotation, and with teacher error. If high-$u$ messages are not where the teacher is wrong, the premise fails, and we will report that as a negative finding about window-based diagnostics.
Check 2: Does the cultural effect exist? The gain from instability-routed supervision over threshold recalibration should grow with how pronoun-dropping (high-context) a language is, as measured by the Language Context-level Ratio. It should be near zero on the low-context English Reddit anchor. A flat relationship would mean the effect is about conversation, not culture. That would still be useful, but we would narrow the claim accordingly.
References
- Hate Speech and Counter Speech Detection: Conversational Context Does Matter
- Context-Aware Toxicity Detection in Multiplayer Games: Integrating Domain-Adaptive Pretraining and Match Metadata
- Exploring Large Language Models for Hate Speech Detection in Rioplatense Spanish
- AfriHate: A Multilingual Collection of Hate Speech and Abusive Language Datasets for African Languages
- PejorativITy: Disambiguating Pejorative Epithets to Improve Misogyny Detection in Italian Tweets
- SEAHateCheck: Functional Tests for Detecting Hate Speech in Low-Resource Languages of Southeast Asia
- Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias
- CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation
- Towards Adaptive Context Management for Intelligent Conversational Question Answering
- Cross-Cultural Similarity Features for Cross-Lingual Transfer Learning of Pragmatically Motivated Tasks