When annotators disagree, current agreement statistics and label-aggregation models have no way to tell two very different annotators apart. The first holds a stable minority view: for example, they consistently rate a comment as more offensive than the crowd does, and they know when they are unsure. The second has a confidence signal that has stopped tracking their own reliability: they are just as sure when they are guessing as when they are right. Cognitive aging research shows that the second pattern is real and measurable. Older adults make false recognitions at roughly the same rate as young adults but are more confident in them, and they do not show the extra hesitation young adults show before rating confidence in an error (False Recognition in Short-Term Memory – Age-Differences in Confidence). Annotation research shows the first pattern is also real: annotators aged 60 and older systematically rate text as more offensive (When Do Annotator Demographics Matter?). Any method that judges reliability by agreement with the crowd will penalize both annotators in the same way. It will silence the minority perspective and fail to catch the monitoring failure.
We propose to measure each annotator's memory monitoring, meaning how well their stated confidence tracks their own reliability, without reference to the crowd and without ground truth. The reference point is the annotator's own repeated judgments, collected in a later session. Concretely, an annotator's confidence on an item should predict whether they give the same label when the item reappears a week later. We turn this into a per-annotator score using mutual information. We use the score to build an agreement coefficient and an aggregation model that down-weight annotators whose confidence carries no information. Annotators who are consistent but in the minority are left alone. We validate the score against a short laboratory memory task with known correct answers.
Core Claim
An annotator's deviation from consensus can be split into two parts. One is a criterion shift: a stable threshold difference that is reproduced across sessions and is the perspective worth preserving. The other is a monitoring deficit: confidence that does not predict the annotator's own reproducibility. These two parts can be estimated separately from data that standard pipelines already almost collect.
Our measure adapts Dayan's metacognitive mutual information, meta-$I$ (Metacognitive Information Theory). Dayan defines it as the mutual information between whether a single observer was correct and the confidence they reported, for perceptual tasks where correctness is known. We replace correctness, which subjective tasks lack, with self-consistency. For annotator $j$, the score is
$$\text{meta-}I^{\text{self}}_j = I(S_j; C_j),$$
where $S_j \in \{0,1\}$ records whether annotator $j$'s label on an item in session 2 matches their label in session 1, and $C_j$ is their binned confidence in session 1. Dividing by the entropy of $S_j$ gives an efficiency score in $[0,1]$. This mirrors Dayan's meta-$I_2$ and removes the dependence on how often an annotator is consistent overall.
We then use the score in agreement measurement. In a Krippendorff-style coefficient, each pairable label normally enters the observed disagreement $D_o$ with unit weight. In a confidence-weighted variant, the weight is the raw stated confidence $c$. We instead use $w_{ju} = \tilde c_{ju}$, the confidence of annotator $j$ on item $u$ after it has been recalibrated against that annotator's own self-consistency curve. Confidence from an annotator whose confidence predicts nothing collapses toward that annotator's base consistency rate. Confidence from a well-monitoring annotator is kept.
This differs from the closest work in a specific way. Confidence-weighted aggregation and confidence-aware alpha take stated confidence at face value (A Bit of This, a Bit of That). Dawid-Skene-style models judge reliability against the inferred consensus (Reliable Crowdsourcing under the Generalized Dawid-Skene Model). Neither separates the two kinds of annotator described above.
Research Program
Prior work that motivates the gap. Three studies point to the gap. Sikora-Wachowicz et al. find that aging selectively lowers Type 2 sensitivity, the ability to rate errors with lower confidence than correct responses. First-order error rates are unchanged (False Recognition in Short-Term Memory). This is exactly the deficit that consensus-based reliability estimates would confuse with disagreement.
Stahel et al. report a newspaper genre ("Notice") with median annotator confidence of 1.0 alongside one of the lowest agreement scores (A Bit of This, a Bit of That). Confidence and agreement can come apart at scale. Their data cannot say whether this reflects real ambiguity or poor monitoring, because they collected no repeat judgments and no annotator profiles.
The HateXplain disagreement analysis finds that 42.6% of disagreement sits at the hate/offensive boundary. Annotators often highlighted identical rationale spans while choosing different labels (Majority Vote Silences Minority Values). That pattern is the signature of a criterion shift rather than inattention, and our decomposition should recover it.
Prior work that constrains the design. McLachlan et al. found that Alzheimer's patients with delusions were more confident on correct responses: 69.7% versus 43.5% high-confidence correct responses (Increased memory confidence and delusions in Alzheimer's disease). This suggests that some monitoring failure is a general upward shift in confidence rather than an error-specific one. Our measure must therefore separate overall confidence level from confidence informativeness. Mutual information does this by construction, because shifting all confidence upward does not change it. We report mean confidence and meta-$I^{\text{self}}$ separately so that this tension in the literature becomes an empirical question in the annotation setting.
Confound we must handle. Self-consistency is itself partly a memory test. A young annotator may simply remember what they answered the first time. We address this in three ways:
- We separate the sessions by at least one week.
- We interleave repeated items among many new ones.
- In session 2 we ask a one-click "have you seen this item before?" question.
The primary analysis uses only repeated items the annotator does not report recognizing. Recognition accuracy and recognition confidence on that question are a second, cheap memory-monitoring measure in their own right.
What we build. Recruitment is through Prolific, the same pool used for POPQUORN. We recruit age-stratified annotator pools, roughly 150 annotators aged 18–35 and 150 aged 60+. Each annotator labels about 300 items with a confidence rating, and we log response times for both the label and the confidence rating. The items are:
- Ruddit offensiveness items drawn from POPQUORN.
- HateXplain posts oversampled from the hate/offensive boundary (HateXplain).
- A block of SQuAD 2.0 answerability items from POPQUORN's question-answering task, where gold labels exist.
About 20% of items are repeated in session 2.
Each annotator also completes a ten-minute abstract-shape short-term recognition task with confidence ratings. It follows the Sikora-Wachowicz design and includes lure and foil probes. From it we get a laboratory Type 2 sensitivity score with true correctness. A short word-list false-memory task of the Deese-Roediger-McDermott kind (lists of associates of an unpresented word) adds a measure of confident false recognition of meaning-related items.
From these data we estimate each annotator's criterion shift (an ordinal threshold offset relative to the crowd), their meta-$I^{\text{self}}$, their laboratory Type 2 sensitivity, and their confidence-RT slowing on later-inconsistent items. The last of these is the hesitation marker that young but not older adults showed in the laboratory study.
Baselines
Unweighted Krippendorff's alpha with majority-vote aggregation. This is the naive reference point and the standard practice. HateXplain reports an alpha of 0.46 under it (HateXplain). BERT trained on its majority labels drops from about 80% accuracy on agreed posts to about 58% on disagreed posts (Majority Vote Silences Minority Values). We compute it on exactly our labels.
Raw-confidence weighting. This is the Stahel-style approach of folding stated confidence into soft labels and agreement. It is the direct competitor for the claim that confidence needs recalibration. It runs on the same labels and confidences with no extra data.
Dawid-Skene extended with confidence. This is the strongest existing method and the one most likely to win. It is a latent-class model in which each annotator has a confusion matrix conditioned on their stated confidence bin, with item difficulty terms following the generalized Dawid-Skene model (Reliable Crowdsourcing under the Generalized Dawid-Skene Model). It learns from the label data alone how much each annotator's confidence should be trusted. The difference is that it judges against inferred consensus rather than against the annotator's own repeats. We fit it with the same expectation-maximization budget and the same confidence binning. We give it the session-2 labels as additional observations so that it sees all the data our method sees. No reported result was found on Lacuna for Dawid-Skene on HateXplain or POPQUORN, so we will establish this baseline ourselves.
Age-only covariate. This follows Pei and Jurgens's mixed-effects approach, where each annotator's weight or offset depends only on age group (When Do Annotator Demographics Matter?). It tests whether monitoring measures add anything beyond the demographic label.
Crowd-Calibrator-style selective prediction. For downstream use we follow Crowd-Calibrator (Crowd-Calibrator), in which a model abstains when its predictive distribution diverges from the crowd label distribution. We build the crowd distributions under each weighting scheme. No reported result was found on Lacuna for Crowd-Calibrator's exact abstention numbers on HateXplain.
Ablations. Two ablations isolate the key ingredient. The first weights annotators by plain test-retest consistency, ignoring confidence. This asks whether confidence informativeness matters at all. The second replaces meta-$I^{\text{self}}$ with laboratory Type 2 sensitivity. This asks whether an external probe can substitute for in-task repeats.
What counts as beating the baselines. Our method must outperform confidence-extended Dawid-Skene on the decisive checks below, with bootstrap confidence intervals over annotators that exclude zero. The plain-consistency ablation must not match it.
Why not just…?
Why not just fit Dawid-Skene with confidence-conditioned confusion matrices? This is the real threat, because it learns confidence trustworthiness from data. Its reference point is the inferred consensus, however. A consistent older annotator with a higher offensiveness threshold will look like a poorly monitoring annotator, since their confident labels often contradict the majority. Our prediction is that Dawid-Skene assigns lower reliability to 60+ annotators even when their session-to-session consistency matches that of younger annotators, while meta-$I^{\text{self}}$ does not.
If Dawid-Skene's per-annotator reliabilities turn out to correlate highly with meta-$I^{\text{self}}$ and with laboratory Type 2 sensitivity, the consensus reference is harmless. The proposal then reduces to a validation result rather than a new method, and we will report it that way.
Why not just calibrate annotators on gold questions? This works where gold exists, so we include it for the SQuAD block. Offensiveness has no gold answer, though, and "gold" built from majority vote reintroduces the consensus problem. The decisive test is whether calibration on SQuAD gold transfers to the subjective task as well as meta-$I^{\text{self}}$ measured within that task. Monitoring ability may well be task-specific, which would favor in-task repeats.
Why not just use annotator age as a covariate? Age is a proxy that mixes generational values with cognitive change, and individual differences within each age group are large. If age alone explains as much variance in weighting quality as the monitoring measures do, the cognitive layer is unnecessary. The age-only baseline settles this directly.
Why not just weight by test-retest consistency without confidence? Consistency alone cannot tell which individual labels to trust; it only rates annotators as a whole. Confidence informativeness adds item-level information. This is cheap enough that it might suffice, which is why it is an ablation and not a strawman.
Proposed Experiments
Simulation. We first check recovery on synthetic crowds. Simulated annotators have planted criterion shifts and planted Type 2 corruption, meaning confidence is drawn independently of whether they will repeat their label. Our decomposition should recover both parameters. Dawid-Skene should confuse the two. The simulation also gives the number of repeats per annotator needed for a stable estimate.
Human studies. Three tests decide whether the idea holds.
The first is construct validity. Meta-$I^{\text{self}}$ should correlate with laboratory Type 2 sensitivity and with confidence-RT slowing on later-inconsistent items, across annotators and within each age group. We expect lower average meta-$I^{\text{self}}$ in the 60+ group, with mean confidence on correct and incorrect items reported separately to address the error-specific versus general-shift tension.
The second is perspective preservation. On held-out session-2 items, the aggregate built under each weighting is used to predict the label distribution of each age group. The criterion is Jensen-Shannon divergence to the held-out group distribution. Our method should predict the 60+ distribution better than Dawid-Skene does, without hurting prediction of the 18–35 distribution.
The third is the downstream effect on boundary items. On HateXplain boundary items, we train classifiers on each aggregate and measure accuracy on disagreed items against the roughly 58% reference, along with selective-prediction coverage at fixed risk. On the SQuAD block, label recovery against gold should not degrade.
Failure condition. If the monitoring-adjusted coefficient changes neither alpha nor the group-level predictions relative to confidence-extended Dawid-Skene, we will conclude that consensus-based reliability already captures annotator monitoring, and report that.
Framing. We present results as protecting minority perspectives from being discarded as noise, not as screening older annotators out of annotation work.
References
- False Recognition in Short-Term Memory – Age-Differences in Confidence
- When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset
- Metacognitive Information Theory
- A Bit of This, a Bit of That: Building a Genre and Topic Annotated Dataset of Historical Newspaper Articles with Soft Labels and Confidence Scores
- Reliable Crowdsourcing under the Generalized Dawid-Skene Model
- Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain
- Increased memory confidence and delusions in Alzheimer’s disease: a preliminary study
- HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
- Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?