When hospitals pool data to train a model that grades disease severity, they pool two different kinds of disagreement. The first is visual: scanners, cameras and staining protocols make the same tissue look different from one site to the next. The second is about labels: each site's clinicians draw the line between "grade 1" and "grade 2" in slightly different places. Current methods treat the first as domain shift, to be removed by aligning image features across sites. They treat the second, if at all, as random label noise to be filtered out. We argue that the label disagreement is neither random nor visual. It is a per-site shift in where the grade boundaries fall on a shared severity scale, and it can be modeled directly with a few parameters per site.
This matters most when supervision is weak, meaning one grade is given to a whole bag of images: all the endoscopy frames from one patient, or all the tissue patches from one slide. In that setting the patient's grade is set by the worst lesion, so the model must search the bag for the image that justifies the label. If a site habitually grades one step higher, the model is trained to find grade-2 evidence in bags that contain only grade-1 tissue. It learns to highlight the wrong images, and the damage spreads into the shared severity ranking for every site. Our proposal is a weakly supervised ordinal model with one shared image-to-severity scorer and a small set of site-specific grade thresholds learned jointly with it. The main test is whether learning the thresholds during training, rather than recalibrating afterwards, improves which images the model identifies as severe.
Core Claim
Our central hypothesis concerns what site identity can stand in for. In multi-site cohorts the identity of each individual rater is usually unknown, but the site is almost always recorded. Site-level disagreement about grading therefore has a simple and identifiable form: the same ordered grades with shifted cut-points. Absorbing that shift during training keeps it from corrupting instance-level credit assignment. The existing work leaves exactly this gap. Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses adapts a max-severity bag model across hospitals but assumes the target site's labels are noise-free and follow the source site's criteria. The Latent Doctor Model models rater style, but it needs per-rater support sets and works on single patches rather than weakly labeled bags. HASD aligns slide features across hospitals and handles prevalence shift, but it treats the labels at every site as objective. Our model differs from all three in one way: it puts cross-site variation in the label-generating step, as ordered cut-points, and not in the feature space or in an unstructured noise matrix.
Model
A shared encoder and scoring head assign each instance $x_i$ in bag $b$ a scalar severity $f(x_i)$. The bag severity $S_b$ is a smoothed maximum over instances. We use the Maxsoft construction from Navigating the MIL Trade-Off: a hard maximum on the forward pass and log-sum-exp gradients with temperature $\beta$ on the backward pass. We choose it because it reflects the rule that the worst lesion decides the grade, and because that paper reports it holds up in low-data regimes. The observed grade $y_b \in \{0,\dots,K-1\}$ from site $s$ follows a cumulative-link model, which treats the grade as a noisy reading of the latent severity against ordered cut-points:
$$P(y_b \le k \mid b, s) = \sigma\!\left(\frac{\tau_k + \delta_{s,k} - S_b}{\gamma_s}\right).$$
Here $\sigma$ is the logistic function and $\tau_k$ are the shared cut-points. The $\delta_{s,k}$ are site offsets constrained so that each site's cut-points stay ordered, and the offsets are fixed to zero at a reference site. The site-specific scale $\gamma_s$ captures how consistent that site's grading is. Each site adds only $K$ parameters. Acquisition shift is handled separately by a standard site-adversarial term on the instance embeddings.
This separation of two nuisance sources is only partly identifiable, and we say so plainly. Suppose one site's camera makes lesions look milder. The model could then explain that site's labels either through a cut-point shift or through a lower severity for its images. We break this tie with a small set of anchor bags. These are cases graded by a common reference, either a reference panel or re-grading of a few dozen bags by the reference site. We also report how the estimated offsets change as the number of anchors varies.
Research Program
The build is modest. On top of frozen foundation-model patch or frame features, we train the scorer, the Maxsoft pooling, the cut-points and the adversarial branch. We then run three studies. The first injects controlled site-level threshold shifts into data where true instance grades are known, so recovery can be measured exactly. The second uses real expert disagreement: panels of pathologists act as "sites" with different grading habits. The third runs the method at scale on a large grading benchmark to check that it does not hurt when conventions are already consistent.
Baselines
The naive reference point is to pool all sites and train Maxsoft MIL with plain ordinal cross-entropy, ignoring site. It shows how much conventions actually hurt. The strongest weakly supervised ordinal model in the file is SATOMIL, from Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer. It uses $K-1$ aggregator tokens, each asking whether any image in the bag exceeds a given grade. It reports a Quadratic Weighted Kappa of 0.826 on LIMUC, against 0.813 for TransMIL and 0.744 for DSMIL. That number comes from a single-convention setting. We rerun SATOMIL on the same features, splits and injected shifts, with the same hyperparameter-search budget as our model (an equal number of configurations, selected on the same held-out split). Beating it means a higher patient-level QWK against clean labels at every shift strength above zero, and no loss at zero shift.
The strongest cross-site competitor is the weakly supervised adaptation method of Domain Adaptation for Ulcerative Colitis Severity Estimation. It freezes source aggregation tokens and penalizes any instance scored above the patient's maximum grade. It reports 71.4% target accuracy from LIMUC to a private hospital, against 52.1% for ADDA. We reimplement it on our pseudo-site splits. Because its triplet loss treats the target label as a hard ceiling, we expect it to fail in a specific way under upward-shifted conventions, and that is the comparison that matters.
For noise-robust training we run LiNC at the bag level. It learns a trust scalar per sample, clusters those scalars with a mixture model, and raised 50%-noise MedMNIST accuracy from 58.82% to 80.23%. We also run MedSSL from Benchmarking Real-World Medical Image Classification with Noisy Labels, which uses class-specific selection thresholds and reached 62.61% on DRTiD against 48.18% for DISC.
Two ablations isolate the key ingredient. One trains the shared model without site terms and then fits per-site cut-points after training. The other replaces structured cut-points with a free per-site confusion matrix over grades.
Why not just…?
Why not just give each site its own output head? With enough labeled bags per site, separate heads could match our bag-level accuracy. Our model is in effect a constrained version of this. The constraint should matter when a site has few bags, because a head fit on 30 patients overfits while $K$ offsets do not. We sweep the number of labeled bags at the smallest site from 20 to its full size. If separate heads match us everywhere on both bag-level and instance-level metrics, the structural claim fails.
Why not recalibrate thresholds after training? This is the most serious threat, since post-hoc recalibration is cheap and should recover most of the bag-level agreement. Our claim is that it cannot repair the instance ranking. By the time recalibration runs, the scorer has already been trained to find step-up evidence in the wrong images. The decisive measure is image- or patch-level QWK and localization AUC on held-out ground truth. If joint training does not beat post-hoc recalibration there, the proposal reduces to recalibration, and we will report it that way.
Why not treat it as label noise and filter it? LiNC's authors note that it trusts systematic, consistently biased labels early in training. LNMBench shows that loss-based filtering discards hard, rare-grade samples. A site that consistently upgrades borderline cases produces exactly this kind of systematic, grade-dependent pattern. Filtering discards the informative intermediate grades and does not reuse the site's labels on a corrected scale. The injected-shift experiment tests this directly.
Why not just align features across sites? Feature alignment, as in HASD or adversarial training, removes differences in how images look. A site's habit of calling Mayo 1 disease "Mayo 2" involves no difference in the images. We keep the adversarial term, and our ablation that removes the cut-points shows how much of the problem feature alignment leaves behind.
Proposed Experiments
The controlled study uses LIMUC: 11,276 endoscopic images from 564 patients, each image labeled with a Mayo grade from 0 to 3, so true instance and patient grades are known. We split patients into four pseudo-sites. Each receives a per-site color and blur transform to mimic acquisition shift. Each also receives a directional labeling habit: for example, one site upgrades patients whose worst image is Mayo 1 to grade 2 with probability $p$, and another downgrades borderline Mayo 2 patients. We vary $p$ from 0 to 0.5 and cross it with the strength of the acquisition shift, with prevalence differing across sites to test whether prevalence shift is mistaken for a convention shift. Metrics are patient-level QWK against clean labels, image-level QWK, recovery error of the injected offsets, and calibration.
The real-disagreement study uses the ISUP set of 100 prostate biopsy slides graded by 20 pathologists, and Gleason2019, 244 cores annotated by six pathologists. We group pathologists into pseudo-sites by their own grading tendency and give each slide the grade of its assigned group. We evaluate against the panel consensus and, on Gleason2019, against pixel annotations. On Gleason2019 we compare against the distributional fit of the Latent Doctor Model, which reports a KL divergence of 0.461 there, noting that it uses rater identity while we use only group identity.
For scale we train on PANDA, where ILRA-MIL reports a Cohen's kappa of 0.6562. If per-center provenance can be recovered from its metadata, we treat centers as sites; otherwise PANDA serves only as a check that the site terms cost nothing when conventions agree. We also plan SICAPv2 patch annotations to measure localization.
Three checks decide whether the claim holds. First, at $p=0$ our model must not lose to SATOMIL. Second, as $p$ grows, image-level QWK for joint training must pull away from post-hoc recalibration. Third, the learned offsets must track the injected ones as long as a few dozen anchor bags are available. If they drift under camera shift without anchors, we report that as the limit of what site identity alone can identify.
References
- Domain Adaptation for Ulcerative Colitis Severity Estimation Using Patient-Level Diagnoses
- The Latent Doctor Model for Modeling Inter-Observer Variability
- HASD: Hierarchical Adaption for Pathology Slide-Level Domain-Shift
- Navigating the MIL Trade-Off: Flexible Pooling for Whole Slide Image Classification
- Ordinal Multiple-instance Learning for Ulcerative Colitis Severity Estimation with Selective Aggregated Transformer
- LiNC: Lightweight Noise Correction via Adaptive Label Refinement
- Benchmarking real-world medical image classification with noisy labels: Challenges, practice, and outlook