Is the Upgrade Real? Separating Disease Progression from Reader Disagreement in Serial Biopsies

Patients under cancer surveillance, such as men with low-risk prostate cancer or people with Barrett's esophagus (a precancerous change in the lining of the esophagus), have a biopsy every year or two. A pathologist grades each one, and a change in grade often triggers treatment. A recorded upgrade can mean three things. The disease may truly have progressed. The needle may have sampled a different part of the tissue. Or a different pathologist may have read a borderline slide differently. Current longitudinal risk models cannot tell these apart, because they take every recorded grade as truth. Current models of pathologist disagreement cannot tell them apart either, because they look at one slide at a time and never at a patient's history.

We propose a temporal model in which the recorded grade at each visit is a noisy reading of a hidden disease state, not the state itself. The hidden state moves forward over time under constraints that reflect biology: progression is mostly one-way, and its rate depends on elapsed time. Each reading is produced through two separate noisy steps. The first is a sampling step, which links the hidden state to what the slide actually shows. The second is a reader step, which links the slide's appearance to the grade a particular pathologist assigns. Because the model sees the whole sequence, it can do something no single-slide model can. If a "low-grade dysplasia" call is followed by two clearly non-dysplastic biopsies, the model can lower its belief in that earlier call after the fact. If an upgrade appears on a slide that the image model rates as unambiguous, the model treats it as more likely real.

Core Claim

Modeling serial biopsy grades as noisy readings of a hidden, mostly one-way disease process will give better-calibrated progression risk than feeding the same grades into a recurrent survival model. The improvement should be largest exactly where pathologists disagree most. The key ingredient is not soft labels as such. It is that the time structure lets future and past visits re-estimate how reliable each individual reading was.

Formally, let $s_t$ be the hidden ordered disease stage at visit $t$ and $x_t$ the slide representation. Let $y_t$ be the grade recorded by pathologist $r_t$, and let $\Delta t$ be the time since the previous visit. The model factorizes as

$$p(y_{1:T}, s_{1:T} \mid x_{1:T}) = \prod_{t} p(s_t \mid s_{t-1}, \Delta t)\; q(s_t \mid x_t)\; \pi_{r_t}(y_t \mid x_t).$$

Here $p(s_t \mid s_{t-1}, \Delta t)$ is a transition model that allows only small backward probability. The term $q(s_t \mid x_t)$ is an image-conditioned compatibility between slide and stage, and it is where sampling error enters. The term $\pi_{r}(y \mid x)$ is a reader model that predicts which grade pathologist $r$ would assign to this slide. Inference uses standard forward–backward smoothing (combining evidence from earlier and later visits) over the small discrete stage space. The output is the probability of the next true stage and of the time to reaching a threshold stage.

Research Program

Each part of the model reuses an existing method. The reader model comes from The Latent Doctor Model for Modeling Inter-Observer Variability. That model uses a conditional variational autoencoder (a network that learns a latent space from which samples can be drawn) to predict the full distribution of grades a pathologist panel would give a single image. Its slide-conditioned prior already learns that borderline tissue produces more disagreement. We use it unchanged as $\pi_r$, adding a reader embedding where pathologist identity is known. Its prediction entropy tracked panel disagreement with Spearman correlation up to 0.728 on ISUP biopsies, which is the evidence that it captures real ambiguity rather than noise. That paper is entirely cross-sectional. Our contribution is to place it inside a temporal process.

Slide representations $x_t$ come from frozen histopathology foundation models aggregated across patches. A Clinical Benchmark of Foundation Models showed these models transfer to the longitudinal BarrettNET cohort without fine-tuning. It also showed they fail on low- and high-grade dysplasia, the same place humans fail. That second finding motivates our approach: better features alone do not resolve the borderline zone.

The temporal structure builds on Dynamic-DeepHit-Lite, a recurrent survival model over serial PSA, MRI and biopsy grades in prostate active surveillance. That model treats each recorded Gleason grade as truth. We keep its covariate pathway for non-histological inputs such as PSA, which modulate the transition rate. We replace its direct grade input with the noisy-reading structure.

One caveat comes from The Impact of an Inter-rater Bias on Neural Network Training. That paper found that networks trained on one rater amplify that rater's style. This constrains the design: a reader model fit to one institution's pathologists could import their style into the trajectory. We therefore estimate reader models across the full cohort and report results stratified by pathologist.

We will build three things. The first is the model above. The second is a semi-synthetic benchmark with known ground truth. We simulate one-way progression sequences, attach a real Gleason2019 or ISUP image at each stage, and draw the recorded grade from that image's actual panel annotations, assigning a random panelist per visit. This produces realistic sequences in which we know which upgrades are reader noise. The third is a real-data evaluation on BarrettNET (3,528 slides from 790 patients over 10 years) and on prostate surveillance data where slides and reader identity can be obtained. Reader identity per visit is the main data risk. Without it, the reader model falls back to the panel-average distribution and the approach still applies, but more weakly.

Baselines

The naive reference point is a landmarking Cox model on the most recent recorded grade plus clinical covariates. In the prostate surveillance cohort, Dynamic-DeepHit-Lite reports landmarking Cox at a 5-year C-index of 0.75 (±0.08). C-index is the concordance index, which measures how well predicted risks rank patients by when events occur.

The strongest existing temporal method is Dynamic-DeepHit-Lite itself. It reports a C-index of 0.82 (±0.08) at 5-year follow-up and 0.79 (±0.11) for a 3-year window, and we will reimplement it on our cohorts. Those figures come from a tabular cohort, so we will re-run Dynamic-DeepHit-Lite on the same patients, splits and endpoints as our model. Both models will receive identical slide features, and both will get the same hyperparameter search budget. There is no reported result on Lacuna for longitudinal progression forecasting on BarrettNET. The best cross-sectional reference there is the ExCo ensemble from A Clinical Benchmark of Foundation Models, with Cohen's κ of 0.651 for 4-class subtyping. We will use it as the per-visit feature source for all models.

A soft-input baseline is the most important control. It feeds the Latent Doctor Model's per-visit grade distribution and entropy into Dynamic-DeepHit-Lite as extra features. This model has exactly the same information as ours but no hidden-state structure.

Two ablations isolate the time structure. The first removes the one-way constraint on transitions. The second removes backward smoothing, so the model filters forward only and cannot revise past readings.

Beating the strongest method means a statistically significant gain in time-dependent C-index and in calibration error over both Dynamic-DeepHit-Lite and the soft-input baseline. The gain should be concentrated in the subgroup of patients whose sequences contain a borderline-entropy visit. A uniform gain would suggest the benefit comes from capacity rather than noise modeling.

Why not just…?

Why not just feed soft labels or reader-entropy into an existing recurrent survival model? This is the real threat. If the soft-input baseline matches our model, our claim fails and the honest conclusion is that per-visit uncertainty features are enough. Our bet is that such a model cannot retroactively discount an earlier reading using later biopsies, because its recurrent state moves forward only. The no-smoothing ablation and the soft-input baseline test this directly. The semi-synthetic benchmark sharpens the test, because there we can measure whether each model correctly labels individual upgrades as noise.

Why not just use majority vote or label smoothing? Majority vote needs several reads per biopsy, which routine surveillance never provides. Label smoothing applies the same uncertainty to every slide. The Latent Doctor Model showed that majority-vote training discards information about which cases are borderline and becomes overconfident on them. We include a smoothed-label Dynamic-DeepHit-Lite as a further control.

Why not just replace pathologist grades with an AI grader at every visit? Automated graders can beat the average pathologist on prostatectomies, as shown by Development and validation of a deep learning algorithm for improving Gleason scoring (accuracy 0.70 versus 0.61). But they inherit the style of their annotators, as The Impact of an Inter-rater Bias on Neural Network Training shows, and they still fail on dysplasia, as the BarrettNET benchmark shows. An AI grader also does nothing about sampling error. We run this as a baseline: AI grades as hard inputs to Dynamic-DeepHit-Lite.

Why not just use a Cox model, given small cohorts? The prostate surveillance results above already show Dynamic-DeepHit-Lite outperforming landmarking Cox. Our hidden-state model also has few parameters on top of the frozen features. It uses a handful of ordered stages and a constrained transition matrix, which keeps overfitting risk comparable to Cox.

Proposed Experiments and Decisive Checks

The first decisive check is on the semi-synthetic benchmark. Using Gleason2019 and ISUP panels, we measure AUROC for classifying each observed grade change as true progression versus reader noise. We also measure how accurately each model recovers the hidden stage at each visit. If the noisy-reading model does not clearly beat the soft-input baseline here, where the assumptions hold by construction, we stop.

The second check is on BarrettNET and prostate surveillance data. We forecast time to a harder endpoint: adenocarcinoma, or high-grade dysplasia confirmed by a review panel, or treatment-triggering progression confirmed on a subsequent biopsy. We report time-dependent C-index, integrated Brier score (a measure of calibrated survival prediction) and calibration slope. Results are stratified by visit entropy and by pathologist.

The third check is prospective and costs some clinician time. Take biopsies where the model asserts with high confidence that a recorded upgrade is noise, along with matched biopsies where it asserts the upgrade is real. A blinded expert panel re-reads both sets. The model's claims should predict the panel's re-grading. This mirrors the re-staining audit in Clinical utility of receptor status prediction in breast cancer, which used model flags to uncover 31 confirmed misdiagnoses. Agreement in this audit is the clearest evidence that the model separates biology from disagreement rather than simply smoothing the trajectory.