Does Human Disagreement Transfer Across Languages?

When people label irony, hate speech or sentence similarity, they often disagree. Much of that disagreement is not noise. Some items are genuinely ambiguous, and different groups of people (older and younger readers, for instance) draw the line in different places. A line of work called "learning from disagreement" trains models to predict the whole spread of human labels rather than one majority answer. A separate line of work, zero-shot cross-lingual transfer, fine-tunes a multilingual model in one language (usually English) and applies it unchanged to others. Nobody has asked whether the first kind of model survives the second kind of move. If a model learns from English annotators how contested an ironic reply is, does it correctly predict how contested the same kind of reply will be for Spanish or Hindi readers? Or does it carry English annotators' cultural thresholds into a population that does not share them?

We propose to answer this on MultiPICo. It is the one resource in reach with disaggregated irony labels, meaning every individual annotator's label is kept, along with annotator demographics, across nine languages. The central move is to split predicted disagreement into two parts. The first part comes from the item itself: some posts are simply hard to read. The second part comes from groups that systematically diverge. We then measure separately how well each part transfers. We also test a cheap repair that uses no target-language task labels. The model keeps the per-group predictions it learned in English and swaps in the demographic make-up of the target annotator pool.

Core Claim

Our hypothesis is that zero-shot transfer carries item-intrinsic ambiguity well, because multilingual encoders align meaning across languages (UDify; Zero-shot Reading Comprehension). We expect it to transfer group-driven disagreement poorly, both in size and in direction, because that disagreement depends on who is annotating (Reconsidering Annotator Disagreement about Racist Language found annotator race interacting with 7 of 12 topics). The prediction has a concrete, falsifiable form. A group-conditioned model reweighted to the target population's demographics should beat both a single pooled soft-label model and a calibrated hard-label model at predicting target-language label distributions. Its advantage should be largest in languages where MultiPICo's polarization index (its measure of how far demographic groups diverge while staying internally consistent) is high. If the pooled model does just as well, then disagreement is mostly language-universal ambiguity, and the perspectivist machinery is unnecessary across languages. That would also be a useful finding.

Relation to Prior Work

Prior work plays three roles here.

Motivating gaps. MultiPICo motivates the study and supplies the data. It released the disaggregated labels and showed that age and student status polarize irony judgments across languages. However, it tested only prompted large language models, which it found biased toward younger annotators (Spanish Pearson $r = 0.317$ young vs. $0.190$ old). It never trained a multilingual encoder on label distributions or transferred one. Cross-lingual Subjectivity Detection for Resource Lean Languages shares the word "subjectivity" but classifies opinionated versus factual documents with hard labels, so it does not address annotator disagreement. Multilingual and Cross-Lingual Graded Lexical Entailment built graded judgments in four languages but averaged the ratings. It reports high cross-lingual agreement on typicality, which is evidence that some ambiguity is shared across languages. That is precisely the component we expect to transfer.

Supplied methods. We reuse several techniques. Soft-label training comes from Learning from Disagreement: A Survey, where soft labels reached 68.46% vs. 65.57% accuracy on CIFAR-10H. Distribution-aware losses and heads come from Beyond Averages. Subpopulation-specific latent labels come from NUTMEG, and parametric per-perspective adapters from Hypernetworks for Perspectivist Adaptation. All four are monolingual, and none tests a perspective model on annotators from another language.

Constraints. Majority Vote Silences Minority Values shows that majority-vote models lose 22–28 accuracy points on contested items and remain overconfident (0.710 confidence on errors). Separately, Automatic Prosody Annotation shows that enforced label consistency can beat human labels when the end use is speech generation. We take the second result as a reminder to score our models against the target human distribution itself, not against an aesthetic preference for consistency.

Research Program

The model is an XLM-RoBERTa encoder with a small head that outputs $p_\theta(y \mid x, g)$, the probability that annotators in demographic group $g$ label item $x$ ironic. The groups are age cohort crossed with student status, the two factors MultiPICo found most polarizing. The population-level prediction for a language $\ell$ is

$$p_\ell(y \mid x) = \sum_{g} \pi_{\ell,g}\, p_\theta(y \mid x, g),$$

where $\pi_{\ell,g}$ is the share of group $g$ among language $\ell$'s annotators.

During training on English, the head is fit to each annotator's label under that annotator's group, with soft cross-entropy. Following NUTMEG, an annotator-reliability term down-weights erratic raters. At transfer time we compare two settings. In the "source-mixture" setting, $\pi$ is taken from the English pool. In the "target-mixture" setting, $\pi$ is read from the target pool's demographic metadata. The second setting uses no target task labels, so the transfer stays zero-shot.

The analysis then decomposes each target item's observed disagreement into a within-group part (the average label entropy inside groups, which reflects item ambiguity) and a between-group part (the spread of group means, which reflects polarization). For each language, it reports how well the transferred model recovers each part.

We run three regimes:

  • English-only source. Train on English, test on each of the other eight languages.
  • Leave-one-language-out. Train on eight languages, test on the ninth, which separates "the source is English" from "the source is one language."
  • Variety-level evaluation. Test separately on MultiPICo's 25 regional varieties, which probes the MLQA finding that transfer degrades away from mainstream text.

At roughly five annotators per item (about 94,000 annotations over 18,778 items), empirical distributions are noisy. We therefore estimate a noise ceiling for every metric by splitting each item's annotators into random halves and asking how well one half predicts the other. All results are reported as a fraction of that ceiling.

Baselines

Do-nothing reference. The naive reference point predicts, for every target item, the target language's overall base rate of "ironic." Any model has to beat it on distributional fit, or else it is learning nothing item-specific about disagreement.

Calibrated hard-label transfer. This is the strongest simple competitor. An XLM-RoBERTa classifier is trained on English majority-vote labels, temperature-scaled on an English development set, and its predicted probability is read as the label distribution. It is strong because the item-ambiguity component may already be captured by confidence alone.

Pooled soft-label transfer. The same encoder is trained on the pooled English label distribution with no groups. This is the direct import of Beyond Averages and the survey methods into the cross-lingual setting.

Prompted language models. We run the zero-shot and perspective-taking prompts from MultiPICo, and also elicit verbalized probabilities. The only reported numbers we have are hard-label: ChatGPT zero-shot F1 ranges from 0.152 (Dutch) to 0.567 (Arabic) in MultiPICo. We found no reported result on Lacuna for distribution-level prediction of irony disagreement on MultiPICo, in any language. That metric therefore has to be established here.

Translate-train soft labels. English items are machine-translated into each target language, keep their English label distributions, and are used for training.

Ablation of the key ingredient. The ablation compares the group-conditioned model under source-mixture and target-mixture weights. The two share identical parameters, so any difference isolates the population reweighting.

Fairness and the success criterion. All trained systems use the same encoder, the same English training items, the same number of hyperparameter trials, and English-only model selection. Beating the field means two things. The target-mixture model must achieve lower Jensen–Shannon divergence to the target distributions than the calibrated hard-label and pooled soft-label baselines, with paired bootstrap confidence intervals over items that exclude zero, in at least most target languages. It must also achieve higher correlation with the between-group gap.

Why not just…?

Why not just use a calibrated classifier's confidence as the disagreement estimate?

This is the real threat. Confidence after temperature scaling may track item ambiguity as well as any soft-label head does. Our decomposition settles it. On the within-group component we expect the calibrated classifier to be close to the soft-label models. On the between-group component it should fail, because a single probability cannot express that two groups disagree in opposite directions. If it matches us on both components, we will report that disagreement transfer reduces to calibration transfer.

Why not just translate the English training data?

Cross-lingual Transfer or Machine Translation? found that direct transfer matched or beat translated data for Japanese and Korean similarity scoring (81.02 vs. 76.12 on JSICK-STS against LaBSE). However, that result concerns averaged scores. For irony there is an extra problem: translation keeps the English annotators' distribution attached to text that a target reader may judge differently. Translate-train is included as a baseline. Our prediction is that it helps the within-group component and does nothing for the between-group component.

Why not just prompt a large language model with demographic personas?

MultiPICo found trait-based persona prompts largely ineffective. It also found that open multilingual models such as mT0 and PolyLM were near chance on non-mainstream varieties, and that ChatGPT's tilt toward young annotators persisted. We still run persona prompting with verbalized probabilities, because a stronger prompting protocol could close the gap. If it matches the target-mixture model on distribution metrics, that is evidence the parametric group head is not needed.

Proposed Experiments and Decisive Checks

The primary metrics are three:

  • Distribution fit: per-item Jensen–Shannon divergence and cross-entropy against target empirical distributions.
  • Disagreement ranking: Spearman correlation between predicted and observed disagreement.
  • Group-gap correlation: per-item correlation between the predicted and observed difference between young and old annotators' "ironic" rates.

Each metric is normalized by the split-half ceiling.

Three checks would be decisive:

  1. Does reweighting help? Target-mixture must beat source-mixture on distribution fit. If it does not, the population composition is irrelevant, and the demographic story fails across languages.
  2. Is the advantage driven by polarization? Across the nine languages, the target-mixture model's gain over the pooled soft-label model should correlate positively with MultiPICo's per-language polarization index. A flat relationship would mean the gain is not driven by polarization.
  3. Does the group gap transfer? The group-gap correlation must be clearly above zero in target languages. If English-learned age effects point the wrong way in some languages, as the cultural-projection worry predicts, we will identify those languages and report them as the negative finding they are.

As a secondary use, predicted disagreement can stand in for the test-time disagreement tercile that the slice-wise calibration in Beyond Averages assumes is known. We will check whether this recovers part of that method's benefit without oracle information.