Spending Delayed Ground Truth on Past Disagreements

When several pathologists grade the same tissue and disagree, today's pipelines treat the disagreement as noise. They take a majority vote, fit a model of each rater's bias, or flag the case for an expensive confirmatory test. Months later, definitive answers often arrive for a few of those cases: an immunostain, an expert adjudication, a surgical specimen. They are used once, to settle the case they belong to. This proposal argues that this wastes them. A single delayed answer on a case where two raters collided can reveal how both raters err in that kind of tissue. Because every past read those raters made on similar tissue is still stored, that one answer can retroactively correct many old diagnoses at once.

The idea is borrowed, as a design principle and not as literal algebra, from wireless communication. There, "retrospective interference alignment" shows that channel knowledge arriving too late to prevent interference is still valuable. Receivers keep their corrupted past signals, and the transmitter later sends one combined "remedy" message that cleans up several receivers' stored signals simultaneously (Completely Stale Transmitter Channel State Information is Still Very Useful). The pathology counterpart has three parts. Keep the full log of discordant reads. Learn each rater's bias as a function of tissue appearance. Choose which few cases receive delayed ground truth by how many stored reads each answer will correct, rather than by how uncertain that one case is.

Core Claim

Given a fixed budget of delayed ground-truth labels, the claim is that two choices together correct more of a stored archive of multi-rater reads, measured against expert truth, than the same budget spent at random, by per-case uncertainty, or by generic information gain on a rater model that ignores tissue type. The two choices are, first, to spend the budget on cases where raters disagreed, weighted by how many archived reads those raters made on similar tissue, and second, to re-correct the entire archive after each answer arrives.

The measurable object is the accuracy of the corrected archive as a function of budget. If the claim holds, the gain over the alternatives should be largest exactly where The CrowdGleason dataset located the weakness of current crowd models: subtle sub-patterns such as cribriform glands, where rater error is concentrated and not uniform.

Where the analogy holds, and where it does not

Two wireless results supply the intuition. In the two-user scheme of Completely Stale Transmitter Channel State Information is Still Very Useful, a single "order-2" symbol is useful to both receivers because each already holds the other half of the puzzle. This is why stale feedback lifts throughput from 1 to 4/3. Communication Through Collisions generalizes this to packet networks: overheard and colliding receptions are queued rather than discarded, which reaches a sum throughput of 0.9 against 0.5 for scheduled one-at-a-time transmission. The pathology translation is that a case where rater A called Gleason 3 and rater B called Gleason 4 is an "order-2" case. Its resolution informs both raters' bias in that tissue region. A case read by a single rater, or one where all raters agreed, informs at most one rater, or none.

The analogy breaks in three places, and the proposal is designed around those breaks. Labels are discrete, so there is no exact subtraction: correction happens through re-estimated, feature-dependent confusion probabilities. Rater bias is not known channel state but must itself be estimated. And no theorem guarantees a 4/3-style gain. Whether a gain exists, and how large, is the empirical question.

The hybrid-feedback result of On the Degrees-of-freedom of the 3-user MISO Broadcast Channel with Hybrid CSIT does map onto pathology. In that work, mixing some users with instant channel knowledge and others with delayed knowledge reaches 9/5 or 9/4 degrees of freedom. The pathology counterpart is a cheap, instant source of feedback (a deterministic slide-level model) alongside slow, definitive ones. The proposal treats that model's embedding as the coordinate system in which rater bias varies. It does not treat the model as truth.

Research Program

The system has three components.

The first is a stored archive of every read by every rater, with no aggregation applied.

The second is a rater model whose confusion matrix depends on tissue appearance. Concretely, each rater $r$ has a confusion matrix $\Pi_r(c)$ for each of a small number of tissue clusters $c$, defined in the embedding of a frozen patch encoder. The matrices are tied across clusters by a hierarchical prior so that sparse clusters borrow strength. This extends the feature-independent confusion matrices of SVGPCR in The CrowdGleason dataset, which that paper itself identifies as a limitation. It also differs from the per-rater latent Gaussians of Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation and the support-set latent of The Latent Doctor Model for Modeling Inter-Observer Variability. In those two papers, the rater variable conditions a forward generator. Here it is a tissue-dependent quantity that is repeatedly re-estimated and then run backward over the archive.

The third component is a selection rule for the delayed-truth budget. The retrospective value of resolving case $j$ is

$$V(j) = \sum_{(i,r)\in\mathcal{A}} \Big( \Pr[\hat y_{ir} \neq y_i] - \mathbb{E}_{y_j}\Pr[\hat y_{ir} \neq y_i \mid y_j] \Big),$$

where $\mathcal{A}$ is the archive of (case $i$, rater $r$) reads, $\hat y_{ir}$ is the corrected label the system assigns to that read, $y_i$ is the true label, and the expectation is taken over the model's current belief about the unknown truth $y_j$. Computing $V$ exactly requires refitting the model once per candidate case. The proposal therefore also tests a cheap "collision" approximation. For each pair of raters that disagree on $j$, it multiplies the number of archived reads those two raters made in $j$'s tissue cluster by that cluster's current posterior uncertainty about their confusion matrices, and then sums these products over the disagreeing pairs.

Both rules optimize the expected reduction in archive error, which is a standard style of active-learning objective. The proposal does not claim that style of objective as its contribution. The contribution is the problem setting, in which delayed truth corrects stored past reads rather than training a classifier, together with the tissue-dependent rater model that makes collisions informative, and a test of whether the cheap collision rule recovers most of the value of the exact one.

After each batch of delayed labels arrives, the rater model is refit and every archived read is re-corrected. A downstream patch classifier can then be trained on the corrected archive, which allows comparison with published numbers.

Baselines

The naive reference point is majority vote over each case's archived reads, with no delayed truth used. The CrowdGleason dataset reports a quadratic Cohen's kappa of 0.6576 for majority-vote training on CrowdGleason, and 0.7048 for SVGPCR, which models rater confusion without expert labels. Both numbers anchor the "no delayed truth" end of every budget curve.

The strongest existing method is SVGPMIX from the same paper. It mixes resident crowd labels with a slice of expert labels and reports quadratic kappa 0.7276 on the CrowdGleason test set and 0.7814 on SICAPv2 with 10% expert labels. Its expert slice is chosen without regard to which cases were disagreements. That makes it exactly the competitor that tests whether choosing which cases to resolve matters, rather than simply having expert labels at all.

All methods will be run on the same CrowdGleason splits with the same frozen encoder for the downstream classifier. Each method gets the same delayed-label budget, swept from 1% to 10% of cases. Hyperparameters are tuned on the same validation split with an equal search budget. Beating SVGPMIX means one of two things: exceeding 0.7276 quadratic kappa on the CrowdGleason test set at the 10% budget, or matching it with materially fewer expert labels. The proposal also commits to reporting archive-correction accuracy, meaning the fraction of resident reads whose corrected label equals the expert label, because that is the quantity the method directly optimizes.

On Gleason2019, which has six pathologists, Pionono's unweighted kappa of 0.758 (Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation) and the Conditional Latent Doctor Model's KL divergence of 0.461 (The Latent Doctor Model for Modeling Inter-Observer Variability) are the reference points for consensus quality without delayed truth. There, the held-out rater plays the role of the delayed source.

Two ablations isolate the key ingredients. The first keeps the tissue-dependent rater model but chooses cases at random. This isolates the value of selection. The second keeps collision-targeted selection but uses a single, feature-independent confusion matrix per rater. This isolates the value of tissue dependence, and the hypothesis predicts that collisions lose most of their value in this setting.

Why not just…?

Why not just refit SVGPMIX whenever expert labels arrive? That is the "random selection" ablation run with a feature-independent model. It is a real competitor, and if it matches the full method at every budget, the claim fails. The evidence suggesting it will fall short comes from its own authors: The CrowdGleason dataset reports that resident errors concentrate in subtle patterns, which a uniform confusion matrix cannot represent. Expert labels spent on easy large-gland cases therefore correct little. The per-cluster breakdown of archive accuracy will show directly whether that is where the gap opens.

Why not just send the most uncertain cases for confirmatory testing? Uncertainty sampling ranks a case by how unsure the model is about that case alone. It ignores how many other archived reads the answer would fix. The two rankings coincide when uncertainty is spread evenly across raters and tissue, and they diverge when a few raters dominate a cluster. The budget curves will show whether that divergence matters on CrowdGleason's seven residents. If it does not, the conclusion is that the archive structure carries no extra signal there.

Why not just use generic information gain on a rater model? This is the most serious threat. Information gain about rater parameters, computed on the tissue-dependent model, is close to the exact objective $V$. The honest position is that the proposal's selection idea is a specific, cheap instance of this objective, so the proposal does not claim to beat it. Instead, it tests two things. The first is whether information gain on a feature-independent model, which is the version a practitioner would build on top of existing crowd models, loses to the collision rule on the tissue-dependent model. The second is whether the cheap collision rule retains most of exact $V$'s gain at a small fraction of its compute. If exact information gain on the tissue-dependent model is both affordable and clearly better, that will be reported as the recommendation, and the collision rule will be presented only as an approximation.

Why not just replace human reads with a deterministic slide model? Clinical utility of receptor status prediction in breast cancer and misdiagnosis identification using deep learning on hematoxylin and eosin-stained slides shows such a model can reach PPV 0.9992 on its most confident 30% of cases and catch 31 human misdiagnoses. That confident subset leaves most cases to humans, and subtle grades remain weak points for such models. The proposal uses the slide model to define tissue regions and includes "trust the encoder's own classifier" as an additional reference curve.

Proposed Experiments

The primary experiment uses CrowdGleason. Resident reads form the archive, and expert labels are released in budgeted batches as simulated delayed truth. Budget curves are reported for archive-correction accuracy and for downstream quadratic kappa on the CrowdGleason test set and SICAPv2, over five seeds, with paired bootstrap intervals.

The decisive check has three conditions. Collision-targeted selection with tissue-dependent correction must beat SVGPMIX's expert-label usage at matched budget. It must beat random and uncertainty selection by more than the seed variance. And the feature-independent ablation must lose most of that margin. If the margin survives without tissue dependence, the mechanism is not the one proposed. If no selection rule beats random, retrospective selection has no value on this data.

The secondary experiment uses Gleason2019 in a leave-one-pathologist-out protocol. Each held-out pathologist acts as the delayed source, and the measure is how quickly the remaining five raters' archived reads converge toward that pathologist's labels. This tests transfer to a panel of experienced raters rather than residents.

A diagnostic analysis plots, for each budget, how many archived reads each resolved case actually changed. This directly tests the one-answer-corrects-many premise. It is the pathology analogue of asking whether the remedy symbol really served two receivers. Temporal rater drift, which the file identifies as an open problem, would require read timestamps, which the listed datasets do not appear to provide. It is left out of scope rather than simulated.