Debate for the Annotator, Not the Label: Multi-Label Dialect Annotation

Many sentences written in a regional variety of a language are just as natural in several neighbouring varieties. Over half of dialectal Arabic sentences are valid in more than one region, and 70% of one shared-task development set was judged valid in two or more dialects. Datasets that force each sentence into a single "correct" dialect are therefore wrong by construction. Fixing them requires native speakers to mark every variety in which a sentence is valid. That work is slow, and it is skewed by where each annotator comes from.

The obvious move is to have large language models (LLMs) do more of the work. One option is to let several of them argue it out, a setup called multi-agent debate in which model instances with different assigned roles critique each other over several rounds. Another is to have them reason step by step before answering. We argue this is backwards for this task. Judging whether a sentence "sounds like my dialect" is an intuitive, perceptual judgment, which is exactly the kind of task where explicit deliberation is known to hurt. Debate also pushes agents toward a single consensus answer, which is the opposite of a multi-label target. Our proposal keeps the label decision fast and deliberation-free. The cheap disagreement among per-dialect judges decides which sentence–dialect pairs a human should look at. Debate is used only to produce evidence for and against validity, and that evidence is shown to the human reviewer rather than turned into a label.

Core Claim

We make three testable claims about multi-label dialect validity, the task of deciding for each sentence and each dialect whether a native speaker of that dialect would accept the sentence.

The first claim is about what goes wrong. Chain-of-thought prompting (making the model write out reasoning before answering), reasoning-mode models, and symmetric persona debate will all lower macro F1 relative to direct per-dialect scoring. The loss should come through one specific mechanism, which we call overlap collapse: the predicted number of valid dialects per sentence shrinks below the human count. Deliberating models should also over-rely on supposed dialect keywords that turn out to be shared across regions, so-called false friends.

The second claim is about where human effort should go. Simple disagreement among independent, non-deliberating per-dialect judges should flag the cells humans would correct at least as well as two alternatives: classifier-margin signals of the kind used in active error detection, and "debate deadlock," meaning cases where debating agents fail to agree.

The third claim is about what to show the human. Short for-and-against evidence spans generated by a two-agent debate should make native-speaker review faster and more accurate than either a bare model draft or no draft. They should also reduce how often reviewers accept wrong drafts and narrow the disagreement between annotators from different regions.

How This Sits Against Prior Work

The motivating evidence comes from dialectology. Revisiting Common Assumptions about Arabic Dialects in NLP built MLADI, in which 33 native speakers from 11 countries judged the validity of 978 sentences. That study found 56% of sentences valid in multiple regions. It also showed that keyword-based identification falls into the false-friend trap, with high precision and very low recall. The NADI 2024 shared task reports 70% multi-dialect validity on its development set and a best multi-label macro F1 of only 50.57. A Gold Standard with Silver Linings and Common Ground, Diverse Roots find the same phenomenon in Bosnian/Croatian/Montenegrin/Serbian and in Spanish varieties. All four papers treat human annotation as static, one-pass work. None places a model in the annotation loop, and none tests LLM reasoning or debate.

The constraining evidence comes from work on reasoning-induced degradation. Mind Your Step (by Step) shows that chain-of-thought hurts on implicit statistical learning, with a 36.3% absolute drop for o1-preview against direct GPT-4o. It also shows that deliberation creates a bias toward general rules that rationalizes away exceptions. Multi-label dialect validity is close to both conditions: speakers know intuitively what is acceptable, and the data is full of exceptions to rules. Rationales Are Not Silver Bullets finds that rationales hurt on 11 of 18 tasks, mostly because incorrect reasoning contaminated the answer. Evaluating Large Language Models as Expert Annotators finds that chain-of-thought hurts guideline-bound annotation and that reasoning models become stubborn in discussion. Those discussions fall short of the oracle that would pick any agent's correct answer. These results argue against putting deliberation on the label path. They do not show whether deliberation is useful for a human reader, and that is the question we ask.

The method sources come from debate and active annotation. LASE gates debate on token-level confidence and uses a leader–supporter structure. On GPQA-Diamond, ungated debate (62.1%) falls below majority voting (62.7%) while LASE reaches 65.2%. FORD structures debate as stance, rebuttal, and concession, and it documents that weaker agents distract stronger ones. Both evaluate single-label English reasoning tasks, where consensus is the right target. We reuse their mechanics, but we change the debate's goal from reaching a verdict to producing evidence. ActiveAED shows that iteratively sending margin-flagged items to humans beats static error detection, raising average precision from 80.3% to 86.6% on SI-Flights. It does so at roughly 1,000 GPU hours of retraining. We compare its kind of signal against disagreement among cheap LLM judges, which needs no retraining. ELYADATA supplies the strongest automatic system for comparison: a static ensemble of fine-tuned Arabic encoders trained on vocabulary-expanded multi-labels.

Research Program

The system has three parts.

The first part is a panel of per-dialect judges. For each sentence and each candidate dialect, one LLM call answers a yes/no question with no reasoning: would a speaker of dialect $d$ naturally produce this sentence? The call is framed with a regional persona, and we read the probability of "yes." The calls are independent and are never cast as "which dialect is this," so there is no zero-sum pressure between dialects. A fine-tuned encoder with independent per-label sigmoid outputs (rather than a softmax over dialects) is run alongside the panel as a second source of scores.

The second part is a routing rule. A sentence–dialect cell is sent to a human when the judges' "yes" probabilities near the decision threshold disagree, or when the panel and the encoder disagree with each other. Cells that are unanimous and confident are accepted automatically.

The third part is the evidence generator. Only for routed cells, two agents run a short gated debate in the style of LASE. One agent defends validity in dialect $d$, the other argues against it, and each must cite a specific span of the sentence and name the feature it relies on (a word, a particle, a form, a spelling, a register). The output is the pair of spans, shown to the reviewer next to the sentence. The debate never outputs a label.

Human corrections serve two purposes. They update the encoder, but in cheap batched rounds rather than ActiveAED's per-iteration retraining. They also supply new few-shot examples for the judges.

We study the components separately first, using existing gold data to simulate humans, and then run a controlled annotation study with native speakers.

Baselines

The naive reference points are a single direct zero-shot prompt that asks the model to list all valid dialects, and a single-label argmax. The argmax shows how much the multi-label framing alone buys.

The strongest automatic baseline is the ELYADATA system. It is a max-probabilities ensemble of fine-tuned MarBERTv2-family encoders trained on lexically transformed multi-labels. It won NADI 2024 Subtask 1 with test macro F1 50.57 and reported development-set macro F1 of 0.580, according to NADI 2024 and ELYADATA. We re-train it on the same training data our encoder uses. We beat it on labels if our judge panel combined with the encoder exceeds 0.580 macro F1 on the NADI development set at matched label budget. We beat it on annotation if the human-in-the-loop pipeline reaches a target F1 on MLADI-style gold with fewer human judgments.

For BCMS, the reference is the NB Web system from A Gold Standard with Silver Linings, at 88.5% single-label accuracy and 91.0% permissive multi-label accuracy, with native-speaker human accuracy of 94.3%. Approaching the human figure is the meaningful target. For CUBANSPVARIETY, the reference is the fine-tuned BETO result of 78.69% multi-class accuracy from Common Ground, Diverse Roots. For MLADI we found no reported model result on Lacuna, so we establish one.

The deliberative baselines all use the same underlying model: chain-of-thought prompting, reasoning-mode inference, self-consistency voting over sampled answers, symmetric persona debate in the FORD style, and LASE-gated debate. All of them see the same prompts and exemplars, and we match their token budgets where the comparison concerns accuracy per unit of compute.

For acquisition, the baselines are ActiveAED-style margin ranking on the encoder, the DMmean-pred training-dynamics signal from Common Ground, Diverse Roots, debate deadlock, and random sampling. All rank the same pool of cells and are scored on the same simulated budgets.

Three ablations isolate our ingredients. The first swaps our independent per-dialect judging framing for a "which dialects" framing. The second compares evidence spans against a bare draft label. The third compares evidence spans against encoder token-attribution highlights, which tests whether debate adds anything beyond a cheap explanation.

Why not just…?

Why not just prompt one model directly for all valid dialects? This baseline may well beat every deliberative variant. If it does, that supports our first claim. The question is whether a single list-all prompt under-predicts overlap compared with independent per-dialect yes/no calls, since asking for a list invites the model to pick one or two dialects. The framing ablation answers this directly.

Why not just use self-consistency or committee disagreement for routing, and skip debate? This is the real threat, and we partly accept it: our router is cheap committee disagreement, not debate. Debate earns its place only if its evidence spans improve human review beyond what a draft label or encoder highlights provide. If the human study shows no gain in speed, accuracy, or acceptance of wrong drafts over highlights, we drop debate. The resulting recommendation would be a judge panel plus highlights.

Why not just fine-tune an encoder ensemble and do ActiveAED-style uncertainty sampling? The ELYADATA-style ensemble won NADI, but its 50.57 test F1 leaves substantial room for improvement. Margin signals from a model trained on noisy geolocation labels may also mark shared sentences as confidently single-dialect. Geolocation was only 49.3% accurate for Tunisian in NADI 2024. The acquisition comparison measures this directly. If margin ranking matches judge disagreement, we keep the encoder as the router and avoid LLM calls entirely.

Why not just have native speakers annotate everything without model drafts? Unassisted annotation is the gold-standard procedure. However, it costs roughly linear human time per cell, and it carries regional bias: Moroccan and Saudi annotators differ by 0.29 on the dialectness score for identical sentences in Revisiting Common Assumptions. We include no-draft review as a condition in the human study, so anchoring costs and time savings are measured rather than assumed.

Decisive Experiments

The first experiment tests overlap collapse. On MLADI per-country validity, the NADI 2024 development set, the BCMS gold standard, and CUBANSPVARIETY, we compare all labeling strategies on macro F1, recall on multi-valid sentences, and the cardinality gap. The cardinality gap is

$$\Delta = \frac{1}{N}\sum_{i=1}^{N}\left(|\hat{Y}_i| - |Y_i|\right),$$

where $N$ is the number of sentences, $\hat{Y}_i$ is the predicted set of valid dialects for sentence $i$, and $Y_i$ is the human set. The first claim predicts that $\Delta$ is clearly negative for chain-of-thought and debate and near zero for direct per-dialect judges. The effect should be concentrated in sentences with low dialectness scores, since MLADI reports that dialectness correlates with the number of valid dialects at $r = -0.52$. We also hand-code a sample of errors as false-friend driven or not, checking whether deliberation increases the false-friend share.

The second experiment tests routing. Using held-out gold as a simulated human, we compute the average precision with which each routing signal finds the cells where the draft is wrong. We also trace F1 as a function of human budget for each signal. If judge disagreement does not beat margin and DMmean-pred, the second claim fails.

The third experiment is the human study. It uses native speakers drawn from at least Maghrebi, Levantine, Gulf, and Egyptian backgrounds on routed NADI and MLADI-style cells, plus BCMS speakers if recruitment allows. Each reviewer sees one of four conditions: no draft, draft label, draft with encoder highlights, or draft with debate evidence. We measure time per cell, accuracy against a held-out multi-region panel, the rate at which reviewers accept incorrect drafts, and disagreement between annotators from different regions. The third claim holds only if debate evidence beats highlights on accuracy or acceptance of wrong drafts without costing more time. That outcome would show deliberation is useful when a human reads it, even though it hurts when a model turns it into a label.