Anchored Multi-Site Label Aggregation for Severity-Weighted Hazard Indices

When hospitals or field teams label cases independently, each site's annotators usually see only their own site's cases. Suppose one site reports twice as many critical findings as another. Standard label aggregation cannot tell whether that site truly has more disease or whether its annotators simply use a looser definition. This failure becomes dangerous when the summary that matters is a severity-weighted index, and environmental chemistry has a well-known example. Håkanson's Potential Ecological Risk Index (PERI) divides each pollutant's concentration by a local background level, multiplies by a toxicity weight (30 for cadmium, 1 for zinc), and sums the results. At one Grand Canal site that index reached 475.1, and nearly 80% of it came from a single trace metal. The section-wide mean of 110.9 hid that site entirely (Ecological Risk Assessment of Heavy Metals in Sediment from Beijing-Hangzhou Grand Canal). The lesson carries over to labels. A large weight multiplies the error in whatever term it touches, and in label data the heavily weighted terms are the rare, severe classes, which are exactly where annotator models are least reliable.

We propose a multi-site annotator model built to estimate a severity-weighted, background-normalized hazard index for each site. In this model, each site's annotators share a site-level bias. A small set of shared "anchor" cases plays the role PERI's background reference plays: it separates real differences in prevalence from differences in labeling habit. Anchor cases are allocated to minimize the variance of the weighted index, not to maximize labeling accuracy. Severity weights enter only the estimand and the allocation rule. They never enter the likelihood. That design choice is deliberate, and the experiments test it directly.

Core Claim

Label-aggregation models in the Dawid–Skene tradition infer each item's true label and each annotator's confusion matrix by expectation–maximization (EM): they alternate between guessing true labels and re-estimating how each annotator errs. They identify annotator bias only through co-annotation, meaning annotators who label the same items. When annotator pools are partitioned by site, the co-annotation graph splits into disconnected pieces. Any bias shared by every annotator at a site then cannot be distinguished from a shift in that site's true prevalence. Running EM per site, or pooling sites with independent annotators, does not fix this. Each piece is fit separately, and the shared bias is silently absorbed into the "truth."

Our claim has three parts. First, a hierarchical confusion model with site-level offsets, plus a few dozen expert-labeled anchor items shared across sites, restores identifiability. Second, the anchors should go disproportionately to rare, high-severity classes. Third, under a fixed expert budget, this reduces the error of site-level severity-weighted hazard estimates and of high-risk site flags well below both two-stage aggregation and gold-only estimation.

The estimand for site $s$ is

$$R_s = \sum_c w_c \frac{p_{sc}}{p_{0c}},$$

where $p_{sc}$ is the true prevalence of class $c$ at site $s$, $p_{0c}$ is a reference prevalence (the analogue of PERI's background concentration), and $w_c$ is a fixed clinical severity weight (the analogue of the toxicity factor). A first-order delta-method expansion that ignores cross-class covariance gives

$$\mathrm{Var}(\hat R_s) \approx \sum_c \left(\frac{w_c}{p_{0c}}\right)^2 \mathrm{Var}(\hat p_{sc}).$$

This expression makes the core problem explicit. Rare, severe classes carry squared amplification $(w_c/p_{0c})^2$, and they are also the classes whose confusion parameters have the fewest supporting labels.

Relation to Prior Work

Several papers supply the aggregation machinery but assume a single connected pool of annotators.

  • STABLEVAL uses Dawid–Skene EM to produce fractional "posterior expected credit." It reduces MT-Bench rank standard deviation from 0.259 under majority vote to 0.197. The authors themselves flag sparse overlap and correlated annotator biases as failure modes.
  • Deep Learning from Crowds learns annotator-specific mappings jointly with a classifier.
  • DGPCR pairs deep Gaussian processes with Dirichlet-prior confusion matrices on one breast-pathology cohort.
  • Generalized Dawid–Skene adds item difficulty and shows that adaptive query allocation saves a factor of $\lambda/\lambda_{min}$, the ratio of average to worst-case task difficulty.

We reuse the confusion-matrix EM from the first and the adaptive-allocation idea from the last. The difference is that we change what is allocated (anchor items shared across sites) and what allocation optimizes (variance of $R_s$, not label error). None of these papers models site-shared bias, and all of them treat errors symmetrically across classes.

Two papers on multi-site heterogeneity motivate and constrain the design.

  • Surgical Aggregation handles institutions that label different class subsets by sharing only the relevant classifier heads. It treats each site's labels as ground truth, and its accuracy falls from 89.98% at 2 nodes to 74.73% at 10. We supply the missing part: a model of label noise inside each site.
  • Federated Causal Inference shows that two-stage meta-analysis is less efficient than joint federated estimation. It also shows that baseline site effects bias the estimate unless site indicators are included. We take the analogous position for labels: site offsets are required, and estimation should be joint. Because the EM M-step needs only per-annotator expected count matrices, sites can share those counts instead of raw records.

The broader literature on outcome-definition drift (The Challenge of Heterogeneity in Medical Data Synthesis) gives the real-world mechanism behind site-shared bias. One example is studies requiring two versus three failed cycles before diagnosing repeated implantation failure.

The research file frames a different gap: "toxicity weights inside the aggregation likelihood." We consider that idea mistaken. Bayesian decision theory says the right procedure is to infer posteriors with the correct likelihood and apply costs at decision time. The original argument against this, that post-hoc weighting cannot recover signal majority vote has erased, applies to majority vote only, not to a posterior. We include the weighted-likelihood variant only as an ablation.

Research Program

The model. We build a hierarchical Dawid–Skene model. Annotator $k$ at site $s$ has confusion logits equal to a population base, plus a site offset $\delta_s$, plus an annotator offset $\epsilon_k$. Both offsets get shrinkage priors. Site prevalences $p_{s\cdot}$ get their own Dirichlet priors. Anchor items carry expert labels and are seen by annotators at every site. We fit the model by EM with posterior uncertainty from variational or bootstrap estimates, and we propagate that uncertainty to $\hat R_s$.

Identifiability. We will prove a simple non-identifiability result for disconnected co-annotation graphs. We will then characterize how much information the hierarchical prior alone provides, which is only shrinkage under an assumption that site biases are exchangeable, compared with anchors, which identify the site offset class by class.

Anchor allocation. We will derive an allocation rule that adds anchors where they most reduce the variance expression above. That rule naturally targets classes with high $w_c/p_{0c}$. We will compare it with uniform and accuracy-targeted allocation.

Federated fitting. We will implement one-shot and iterative federated EM that exchanges only expected count statistics, and check empirically that it matches pooled fitting.

Baselines

The naive reference is per-site majority vote with a plug-in $R_s$, which is what a practitioner would compute by default. On LabelMe, majority vote reaches 76.74% accuracy against 82.67% for EM and 83.15% for Crowd Layer (Deep Learning from Crowds). We expect it to be worst on rare classes.

The strongest existing aggregation method is STABLEVAL-style Dawid–Skene EM with posterior expected credit, run per site and then combined into $R_s$ with inverse-variance weights. Its best reported figure is a 0.197 rank standard deviation on MT-Bench (STABLEVAL). No reported result was found on Lacuna for site-level weighted-prevalence error or for high-risk site flagging under any aggregation method, so we will establish those numbers ourselves.

Where item features exist, as in pathology, we add Crowd Layer and DGPCR as feature-aware competitors. The reported pathology F1 scores are 81.79% and 81.91% respectively (DGPCR). We will first reproduce those numbers on our splits.

The gold-only baseline estimates $p_{sc}$ directly from the expert-labeled anchors. It uses exactly the same expert budget as our method and ignores the bulk crowd labels.

Fairness rules apply to every comparison. All methods see the same annotations, the same anchor items, and the same expert-label budget. Every method also gets the same allowance for hyperparameter search, measured in prior or regularization settings, selected on held-out sites.

Three ablations isolate the key ingredients: removing anchors (hierarchical prior only), switching to uniform anchor allocation, and moving the severity weights into the likelihood.

Beating the strongest baseline means a lower root-mean-square error of $\hat R_s$ and a higher area under the curve for flagging the top-decile-risk sites, with confidence intervals that exclude zero across simulation seeds. On real data, it additionally means better coverage of nominal 90% intervals for $R_s$.

Why not just…?

Why not just aggregate with Dawid–Skene and apply severity weights afterward? We do exactly that for the weights. Post-hoc weighting of a correct posterior is the right decision rule. The question is whether the posterior is correct. With site-partitioned annotators it is not, because shared site bias is absorbed into prevalence. The no-anchor ablation and the per-site EM baseline test this. If they match the full model on simulated data that contains site-shared bias, the proposal fails.

Why not run Dawid–Skene within each site and then meta-analyze? Within a site, an annotator bias shared by everyone looks identical to a real prevalence shift, so per-site EM is consistently wrong no matter how many labels the site has. Inverse-variance meta-analysis then combines biased site estimates with falsely tight intervals. This mirrors the site-effect bias that Federated Causal Inference documents for treatment effects. Our simulations sweep the size of the shared bias to show where the two-stage approach breaks down and where it is adequate, which is when the shared bias is near zero.

Why not just have experts label a gold sample at each site and estimate prevalence from it? This is the most serious threat. For common classes with moderate budgets, gold-only estimation may be as good as our method and much simpler. Our advantage should appear only for rare, high-severity classes, where a gold sample of a few dozen items contains almost no positives. In that regime, crowd labels calibrated through the anchors should add a great deal of effective sample size. We report the crossover expert budget per class honestly. If gold-only wins at realistic budgets, the conclusion is to spend money on experts, not on models.

Why not pool everything centrally and add site indicators? Pooling does not create co-annotation. A site indicator in a disconnected graph is confounded with prevalence for the same reason as before. Centralization is also often legally unavailable, so federated count-sharing serves as a practical constraint, not the source of the gain.

Proposed Experiments

Simulation. We simulate 5 to 20 sites with disjoint annotator pools, controlled site-shared bias, class-dependent rarity, and severity weights spanning a 30:1 range, mirroring PERI's cadmium-to-zinc ratio. The ground-truth $R_s$ is known, so we can measure estimation error, interval coverage, and flagging accuracy as functions of anchor budget and allocation rule.

Semi-synthetic pathology. We use the TCGA breast-pathology cohort, which has 20 student annotators and expert ground truth (DGPCR), if its annotator-level labels are obtainable. If not, we use LabelMe (Deep Learning from Crowds). In either case we split annotators into pseudo-sites, resample cases to create different site case mixes, and inject definition drift by remapping labels within a site.

Real cross-site safety labeling. We use IU Chest X-ray and the expert-annotated MIMIC subset as two real sites with partly different label sets (Surgical Aggregation). Several automated labelers act as annotators, each prompted with a site-specific rubric. Severity weights come from a clinician-specified ordering, with critical findings weighted far above incidental ones. ConvAbuse serves as a high-agreement control in which every method should agree (STABLEVAL).

Decisive checks. Three results would settle the main questions:

  1. With site-shared bias present, the anchored hierarchical model's $\hat R_s$ error should fall far below per-site EM, and its 90% intervals should achieve close to nominal coverage while the baselines undercover.
  2. Severity-targeted anchor allocation should beat uniform allocation at equal budget specifically on classes with high $w_c/p_{0c}$.
  3. The crossover budget at which gold-only estimation catches up should be reported per class. That crossover determines whether the method is worth using in practice.