When several pathologists grade the same biopsies, they disagree, and that disagreement carries information about which cases are genuinely ambiguous. In real clinical cohorts, though, no pathologist grades every case. Each slide is seen by a few panel members, so the "panel opinion" on most cases is only partly observed. This proposal treats that sparse panel as a multi-view learning problem. Each rater is a view. Each view is summarized as a graph saying which cases that rater considers diagnostically alike. The graphs are stacked into a three-way array (a tensor) and jointly completed under a low-rank constraint, meaning a small number of shared patterns must explain all of them. The completed tensor predicts what an absent pathologist would have said, and so recovers the full distribution of panel opinions on every case.
The opportunity comes from two literatures that have not met. Multi-view clustering has built mature tools for recovering missing views from a stacked graph tensor and stays stable even when most views are missing. Multi-rater medical learning has shown that keeping individual opinions, rather than a majority vote, improves calibration. But it has mostly assumed every rater is present, or trained each rater's model only on the cases that rater happened to see. The core question is empirical and falsifiable. Does panel disagreement have joint structure across cases and raters that is shared beyond what a per-rater bias vector captures? If so, does exploiting that structure recover absent opinions better than the simpler low-rank models a skeptic would reach for first?
Core Claim
We claim that inter-observer variability is low-rank in a specific multi-way sense. Whether rater $r$ treats cases $i$ and $j$ alike depends on interactions between case type and rater that are shared across raters. One example would be a subgroup of pathologists who merge two Gleason patterns only in cribriform-like tissue. A third-order tensor captures such interactions. Two other representations cannot: a case-by-rater grade matrix, and a fixed per-rater latent code fed to a shared network. If the claim holds, tensor completion should beat both on predicting held-out rater grades. The advantage should be largest at high missingness and on the most-disputed cases.
We also make a second, cautionary claim. A low-rank consensus operator will amplify any bias that all raters share. This includes the "safety-first" over-scoring documented on lung CT. A sparsely observed slice of biopsy-proven truth, added to the same tensor, is the right way to counter that bias.
Research Program
The construction follows the graph-tensor line of multi-view clustering, which supplies the method. Low-Rank Tensor Constrained Multiview Subspace Clustering stacks per-view representations into a tensor and penalizes its tensor nuclear norm, a convex surrogate for multi-way rank. Unified Graph and Low-Rank Tensor Learning for Multi-View Clustering applies a rank penalty based on the tensor singular value decomposition (t-SVD) to stacked affinity graphs. Unified Tensor Framework for Incomplete Multi-view Clustering and Missing-view Inferring and ARSL-IMVC extend this to missing views, reporting stability up to 70–80% missing rates. All of these treat views as objective feature extractors, such as color histograms or text embeddings, and evaluate only on benchmarks like COIL-20 and Caltech. We change what a view is: a human observer with a systematic style and incomplete coverage.
For $N$ cases and $R$ raters, we build an observed tensor $\mathcal{O} \in \mathbb{R}^{N \times N \times (R+1)}$. Slice $r$ holds a grade-agreement score between cases $i$ and $j$ for rater $r$. This score is 1 for identical grades and decreases with ordinal grade distance. It is observed only where rater $r$ graded both cases. The extra slice is a dense similarity graph built from frozen image features. This image slice anchors cases no rater has seen and lets test cases enter transductively. We solve
$$ \min_{\mathcal{A}} \; \|\mathcal{A}\|_{\circledast} + \frac{\lambda}{2}\,\big\| P_\Omega(\mathcal{A} - \mathcal{O}) \big\|_F^2 , $$
where $\mathcal{A}$ is the completed tensor, $\|\cdot\|_{\circledast}$ is the tensor nuclear norm, $P_\Omega$ keeps only observed entries, and $\lambda$ trades off fit against rank. Optimization uses the same alternating scheme (ADMM) as the incomplete multi-view methods above. A rater's predicted grade on an unseen case comes from propagating that rater's observed grades through their completed slice.
Two design issues need care, because a naive transfer would fail on them.
The first is rater order. The standard t-SVD applies a discrete Fourier transform along the third mode, which makes the penalty depend on how the raters are ordered, and raters have no natural order. We therefore compare three options: the standard transform under random rater permutations, a data-derived orthogonal transform fitted to the observed slices, and an order-free Tucker-style penalty. Permutation variance in the results is reported as a diagnostic.
The second is shared bias. The cautionary result in Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction constrains the design. That paper shows radiologist consensus over-scores benign nodules, so a model trained on it reaches sensitivity 0.97 but specificity only 0.23. A consensus-seeking tensor would inherit this. Our remedy follows that paper's use of a small biopsy-proven anchor set. We add a "pathology" slice, observed only on anchor cases, and decompose the completed tensor into a low-rank shared component plus a sparse rater-bias component. The bias component is in the spirit of the sample-specific error term in LT-MSC. The question is whether completing the pathology slice through this decomposition corrects bias better than pairwise metric propagation does.
Computation is bounded by an $N \times N$ singular value decomposition per transformed slice. This is feasible at the DEPP scale of about 1,432 cases. For CrowdGleason's 19,077 patches, it becomes feasible only with anchor-graph approximations, a known scalability limit of t-SVD methods noted in UGLTL.
Baselines
The naive reference is a single classifier trained on majority-vote labels from whatever raters are observed. On DEPP, More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel reports this consensus model at average F1 72.9% and Expected Calibration Error (ECE) 0.183. On CrowdGleason, The CrowdGleason dataset reports majority voting at quadratic kappa 0.6576. Beating it is necessary but not interesting.
The strongest existing competitors come in three families.
The first is the virtual expert panel. It trains one attention-based multiple-instance model per pathologist and averages them, and it reports F1 78.2% and ECE 0.050 on DEPP with full coverage. It is the strongest calibrated panel method on our main dataset. Under sparse coverage, each virtual rater is trained only on that rater's cases. Beating it means lower ECE and higher held-out-rater kappa at matched missing rates, not only at full coverage.
The second family is the shared-latent rater models. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) gives each rater an 8-dimensional Gaussian code feeding a shared decoder, reaching unweighted kappa 0.758 on Gleason 2019. The Latent Doctor Model infers a doctor's style from a support set of that doctor's grades. Its conditional variant reaches KL divergence 0.461 to the expert distribution on Gleason 2019 and an entropy Spearman correlation up to 0.728 on ISUP. These models are the real threat, because a shared decoder fed rater codes is itself a low-rank model of rater behavior. We adapt their classification heads to case-level grading where needed.
The third family is crowdsourcing with learned rater confusion matrices: SVGPCR (kappa 0.7048 on CrowdGleason) and SVGPMIX (0.7276), both from the CrowdGleason paper.
For LIDC-IDRI, the strongest bias-correcting baseline is the Siamese metric propagation from SCH-LND anchors, reported at sensitivity 0.67 and specificity 0.66. For 3k Buds, we found no reported numerical result on Lacuna, so we will run the baselines ourselves.
Fairness rules apply to every method. All methods share the same frozen image features (the same backbone per dataset), the same masking patterns, the same splits, and the same hyperparameter search budget, measured in configurations tried on a validation set of held-out rater entries. The tensor method's several regularization weights are a known weakness of this family, noted for ARSL-IMVC. They get no extra budget.
Two ablations isolate the key ingredient. The first is low-rank completion of the case-by-rater grade matrix with image side-information: the same penalty, minus the third mode and the graph view. The second is our tensor with the cross-rater coupling removed, meaning each slice is completed independently with a matrix nuclear norm. If the full tensor does not beat both ablations, the multi-way claim is false.
Why not just…?
Why not just train on the majority vote? Every multi-rater paper in the file reports that this loses calibration and ambiguity information: ECE 0.183 versus 0.050 on DEPP, and kappa 0.6576 versus 0.7048 on CrowdGleason. Under sparse panels it also hides a confound, since the "majority" on a case depends on which raters happened to be assigned. We include it as the floor, not as a serious contender.
Why not just use Pionono- or Latent-Doctor-style rater embeddings, or confusion matrices as in SVGPCR? This is the alternative most likely to match us. A per-rater code into a shared network already shares statistical strength across raters. It can, in principle, represent interactions between case and rater through its nonlinear decoder. Our argument is about data efficiency, not expressiveness. With 6–20 raters and a few hundred cases, a fixed per-rater code must be estimated from that rater's own cases. The tensor instead borrows each rater's case-pair structure from similar raters, even on cases that rater never saw. The masking-rate sweep settles this. If the shared-latent baseline ties us at every missing rate, we will report that disagreement on these datasets is effectively additive bias, which would itself be a useful finding.
Why not just run low-rank matrix completion on the case-by-rater grade table? This is the simplest low-rank model and is exactly our first ablation. The evidence that third-order structure helps comes only from feature-based multi-view clustering, where LT-MSC gains 5.6–6.2% accuracy over single-view low-rank representation on Yale. That gives no guarantee it transfers to human raters, so the comparison is run directly.
Why not just ensemble independent per-rater models? Under full coverage, the virtual panel's calibration is hard to beat. Under sparse coverage, though, each per-rater model sees a shrinking slice of data, and nothing transfers between models. OMBA-UNet also shows the cost of per-rater branches: 0.34s inference versus 0.02s for a single U-Net. The sweep will show at what missing rate, if any, the ensemble's advantage collapses.
Proposed Experiments
The primary testbed is DEPP (11 pathologists, four Barrett's grades), together with Gleason 2019 (6 pathologists), ISUP (20 pathologists, 100 slides), 3k Buds (9 pathologists), and CrowdGleason (7 residents, using anchor-graph approximation). On datasets with dense annotation, we mask rater-case entries at random at rates from 10% to 80%. We also use a structured pattern in which each case keeps only two or three raters, mimicking real panel assignment. Masked entries serve as ground truth.
The metrics are: per-rater Cohen's kappa on masked entries; KL divergence between predicted and full panel grade distributions; ECE and F1 against all annotations, as in the DEPP paper; and Spearman correlation between predicted and human entropy, as in the Latent Doctor work. Results are reported separately on the top-disagreement tercile of cases.
The decisive check is a two-part comparison. At 50% or more missingness, the full tensor must beat both the matrix-completion ablation and the strongest shared-latent baseline on held-out-rater kappa and panel KL, specifically on the high-disagreement tercile. If it wins only on easy cases, the multi-way claim fails. A secondary check is permutation sensitivity: if results vary substantially with rater order under the standard Fourier transform, the order-free variant must be used and reported as the main method.
The bias experiment pools LIDC-IDRI (up to 4 radiologists, scores 1 to 5) with SCH-LND (180 biopsy-proven nodules), linked through the image slice. It compares three variants: radiologist slices alone, which we predict will reproduce the 0.23-specificity failure; the pathology slice without the sparse bias term; and the full decomposition. The full decomposition must exceed the metric-propagation result of 0.66 specificity at 0.67 sensitivity in order to support the claim that treating truth as one more, very incomplete, view is better than pairwise label propagation. If it does not, we conclude that the low-rank consensus prior is the wrong tool where raters share a bias, and that it should be restricted to settings like pathology panels, where disagreement rather than shared bias dominates.
References
- Low-Rank Tensor Constrained Multiview Subspace Clustering
- Unified Graph and Low-Rank Tensor Learning for Multi-View Clustering
- Unified Tensor Framework for Incomplete Multi-view Clustering and Missing-view Inferring
- Aligning Collaborative View Recovery and Tensorial Subspace Learning via Latent Representation for Incomplete Multi-View Clustering
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- The CrowdGleason dataset: Learning the Gleason grade from crowds and experts
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Latent Doctor Model for Modeling Inter-Observer Variability
- Inter-Rater Uncertainty Quantification in Medical Image Segmentation via Rater-Specific Bayesian Neural Networks