Whose Standard, and When? Age–Period–Cohort Models of Annotator Drift in Medical Labels

When a clinical model degrades over time, the usual story is that the inputs changed: new scanners, new patient mix. A second cause is rarely separated out. The labels also drift, because the humans who make them change how they grade. These changes come from three sources that epidemiologists would recognize. A pathologist grades differently after ten years of practice than after one (an age, or experience, effect). A new grading guideline changes everyone's calls at once (a period effect). Pathologists trained under an older guideline keep applying it for their whole careers (a cohort effect). We propose to model a dataset's labeling behavior with the tool epidemiologists built for exactly this three-way split. That tool is Bayesian age–period–cohort modeling, which fits smooth curves over age, calendar time, and birth cohort and uses smoothing priors to cope with the fact that the three are linked. We put it inside a model of who labeled what, and when.

The payoff is practical. If a shift in label rates is a period effect, it will persist, and old training labels should be corrected toward the new standard. If it is a cohort effect, it will fade as older raters leave, so the right fix differs. A single-annotator dataset collected over years is the extreme case. There the rater's cohort is fixed, and experience and calendar time move in lockstep, so any model that ignores this structure will misattribute drift. Our core claim is that a latent-truth labeling model with age–period–cohort structure on each annotator's grading threshold does two things. It separates label drift from input drift. It also forecasts where the labeling standard is going, and it does so better than static annotator-bias models or naive recency retraining when a guideline or practice change occurs.

What exists and where the gap is

Age–period–cohort analysis rests on the identity cohort = period − age. The exact collinearity means linear trends cannot be uniquely split among the three effects. Bayesian versions replace arbitrary equality constraints with random-walk smoothing priors, which penalize jumps between adjacent ages, years, or cohorts. The Global Burden of Disease analyses in the file use this for forecasting: kidney cancer (Deng et al.) and depression, where a sharp 2020 period effect appears across three countries (cross-national MDD analysis). The file names that literature's own limitation, and it motivates us: these models treat case counts as truth and ignore measurement drift from changing diagnostic thresholds and rater subjectivity. The work closest to ours in method is Wang et al.'s neural discrete-time survival model for credit risk. It joins neural networks to age–period–cohort structure and breaks non-identifiability with regularization plus external macroeconomic calibration. We reuse that idea, with known guideline-change dates playing the part of macroeconomic covariates. We apply the structure to the labelers, not the subjects.

On the annotator side, the evidence that single-rater labels encode personal thresholds is strong. Eight expert pathologists reached full consensus on only 53.5% of melanocytic cases (Haggenmüller et al.). Across 120 pathologists, agreement was modest (Fleiss' kappa 0.43), and pruning low-conformity raters improved models (Variability Matters). With 12 veterinary pathologists, a consensus of three shrank the spread of F1 across training sets from an interquartile range of 0.038 to 0.016 (Wilm et al.). TAB models each annotator's segmentation "style" with learned transformer queries and adds a stochastic head for random error. All of these assume an annotator's bias is fixed in time; the file lists that as an open problem. None asks whether consensus or conformity is itself moving. Work on temporal shift, such as Guo et al.'s evaluation of domain generalization and adaptation under temporal clinical shift and Bordner et al. on scanner changes, treats degradation as a shift in data. It does not decompose the shift in labeling. The file's section on prior combinations was not available to us. Within what we have, no paper models time-varying annotator thresholds with age–period–cohort structure.

Core model

Each case $i$ has an image $x_i$ and an unobserved true ordinal grade driven by a network score $f_\theta(x_i)$. Annotator $j$ entered practice in year $s_j$ (their cohort $c$). They label case $i$ in calendar year $t$ (the period), at experience $a = t - s_j$ (their age). Each annotator applies grade thresholds that shift over time:

$$P(y_{ij} \ge k) = \sigma\big(f_\theta(x_i) - \kappa_k - \alpha_{a} - \beta_{t} - \gamma_{c} - u_j\big),$$

Here $\sigma$ is the logistic function, $\kappa_k$ are shared grade cut-points, $\alpha$, $\beta$, and $\gamma$ are experience, period, and cohort offsets, and $u_j$ is a person-level offset. We give $\alpha$, $\beta$, and $\gamma$ second-order random-walk priors, which penalize changes in slope between adjacent bins. Known guideline dates enter as allowed breakpoints in $\beta$, where the smoothing penalty is relaxed. The network and the labeling model are trained jointly by variational inference. For segmentation we replace the ordinal link with TAB's annotator-query mechanism, so the time-varying offsets modulate each annotator's style vector.

We will be explicit about identification. The linear split among experience, period, and cohort is not identified by data, only by the prior. The curvatures and breaks are identified. These are second differences, meaning accelerations and sudden jumps such as a guideline adoption. So the method reports the identified quantities as its main outputs. It treats the linear split as a sensitivity analysis over a range of prior anchors. A single annotator with fixed cohort gives only one combined experience-plus-period curve. Our model says so and refuses to split that curve without external anchors. We regard this as a feature: static methods silently absorb the curve into the "truth."

Research Program

We will build the joint model and a simulator for labeling drift. The simulator takes any multi-rater dataset, assigns each rater a career start year and annotation dates, and shifts their thresholds according to specified experience, period, and cohort curves. It includes step changes that mimic guideline revisions. We will then answer three questions. First, can the model recover the injected curvatures and breakpoints? Second, does correcting historical labels toward the forecast future standard improve accuracy on later cases? Third, does the decomposition correctly predict whether a shift will fade (cohort) or persist (period)? On real data we will analyze archives in which named raters' grades carry timestamps, fitting the model and checking its breakpoints against documented guideline revisions.

Baselines

The naive reference trains on all historical labels as if they were truth, with a single annotator or majority vote, and tests on later periods. The consensus baseline follows Wilm et al.: train on majority vote of three raters per case. It is the field's standard remedy and is strong against noise that is independent across raters. It should fail when all three raters share a period shift.

The strongest existing annotator-aware method is TAB. It reports Average Soft Dice of 96.70% for optic disc and 85.52% for optic cup on RIGA with six ophthalmologists. We will run TAB with the same ResNet34 backbone, data splits, and hyperparameter search budget as our model. We will first reproduce those numbers on the standard static RIGA split. Beating TAB means two things. On the static split we must lose no more than 0.5 Soft Dice points, which shows the temporal terms cost nothing when there is no drift. On drift-injected splits we must show a significant gain in future-period Dice against held-out consensus.

The temporal-adaptation baseline follows Guo et al.: domain generalization and adaptation methods treat calendar years as domains. No reported result for these methods on annotator-drift benchmarks was found on Lacuna, so we will run them under our own splits and budget.

Three ablations isolate the key ingredient, the structured three-way decomposition. The first gives each annotator-year a free, unsmoothed offset. The second keeps only a period random walk and drops experience and cohort. The third uses first-order rather than second-order smoothing and drops the known-breakpoint anchors.

Why not just…?

Why not just retrain on the most recent labels? Recency retraining handles persistent period shifts if recent data are plentiful, and it is a real threat when they are. It cannot tell a cohort shift from a period shift. It therefore throws away older data that remain valid, and it overreacts when a shift comes from one departing rater. Our experiments vary how much recent data is available. We expect recency retraining to match us when recent data are abundant and to lose when they are scarce, which is the usual clinical case. If it matches us throughout, the decomposition adds nothing for prediction. Its value would then shrink to explanation, and we will report that.

Why not just use a consensus of three raters? Wilm et al. show consensus stabilizes training, but their raters labeled at one time. A guideline change moves all three votes together, so consensus averages away idiosyncrasy but not shared drift. The drift-injected splits with a shared period step test this directly.

Why not just give each annotator a free offset per year? This is the most serious simpler alternative. It is the first ablation above. With many labels per annotator-year it may do as well as our model. The smoothing and cohort structure should matter when annotator-years are sparse and when forecasting beyond observed years, where free offsets have nothing to extrapolate. We will report the crossover point in labels per annotator-year. If none exists in realistic ranges, we will narrow the claim to forecasting.

Proposed Experiments

The semi-synthetic stage uses three datasets with known raters: the 12-pathologist canine mitotic figure data (Wilm et al.), the 120-pathologist cell detection set (Variability Matters), and RIGA via TAB. We inject drift scenarios: pure period steps, cohort persistence, experience-driven learning curves, and mixtures. The key metrics are three. The first is error in recovering injected second differences and breakpoint dates. The second is F1 or Soft Dice on future-period cases against held-out unshifted consensus. The third is calibration, measured as expected calibration error, together with the accuracy of forecast future label rates.

The decisive check is a counterfactual pair of scenarios. One has a cohort-driven shift and the other a period-driven shift, and both produce identical aggregate label-rate curves in the training window. Our model should forecast divergent futures and be right in both. Every baseline, by construction, forecasts the same future.

The real-data stage fits the model to a timestamped institutional grading archive. It asks whether estimated period breakpoints line up with documented guideline dates that were withheld from the prior. A failure here, meaning breaks in the wrong years or none at the guideline change, would falsify the claim that the decomposition reflects real labeling practice rather than prior artifacts.