When several clinicians outline the same organ, they disagree, and the disagreement is informative. Some raters draw consistently larger prostates, some are cautious at ambiguous boundaries, and the boundary of an organ near the apex or base is harder to agree on than at the mid-gland. Current models of this variability either train an ensemble of networks or sample a latent code and run a deep decoder many times at test time. Neither approach guarantees that each sampled outline is a single, connected, organ-shaped region. This proposal builds a segmentation head whose output is a joint Gaussian distribution over every rater's contour. The distribution has two coupled covariances. One runs across raters and captures who tends to agree with whom. The other runs across boundary positions and encodes an anatomical statistical shape model, meaning a Principal Component Analysis (PCA) of how organ outlines vary across patients. The most probable set of contours then solves a Sylvester matrix equation $AX + XB = C$, a linear equation in which one matrix multiplies the unknown from the left and another from the right. This equation can be solved exactly, differentiated exactly, and sampled from exactly, at a cost that grows additively rather than multiplicatively with the number of raters and the number of boundary points.
The practical opportunity is a single network, run once, that produces three things: rater-specific contours, a calibrated spread of plausible outlines, and a guarantee of correct topology by construction. It also supports a task no current method in this literature addresses: given the outlines of two raters on a new scan, predict how the remaining raters would have drawn it.
Core Claim
Represent each rater's outline of a single-component, roughly star-shaped structure (prostate, kidney, optic disc and cup, neonatal brain) as log-radii along $K$ fixed directions from a predicted centre. Stack the outlines of $R$ raters into a matrix $X \in \mathbb{R}^{R \times K}$. Place a matrix-normal prior on $X$ with mean shape $M$, rater covariance $\Sigma_r$ ($R \times R$, learned) and shape covariance $\Sigma_s$ ($K \times K$, initialised from a PCA shape model and optionally refined). A network produces rater-conditioned noisy estimates $F$ with per-rater noise variances collected in a diagonal matrix $D$. Setting the gradient of the negative log posterior to zero and right-multiplying by $\Sigma_s$ gives
$$D\Sigma_r^{-1}X + X\Sigma_s = D\Sigma_r^{-1}M + F\Sigma_s,$$
which is a Sylvester equation with $A = D\Sigma_r^{-1}$ and $B = \Sigma_s$. Both matrices have strictly positive eigenvalues, so the spectra of $A$ and $-B$ cannot overlap. The equation always has a unique solution, and its conditioning is bounded by the smallest eigenvalues of the two covariances, which we control through a diagonal floor. This sidesteps the separation problem that is the main numerical weakness of Sylvester methods in general.
The same structure makes the full posterior tractable. Let $D^{1/2}\Sigma_r^{-1}D^{1/2} = P\Lambda P^\top$ and $\Sigma_s^{-1} = Q\Gamma Q^\top$ be eigendecompositions. In the rotated and rescaled basis, the posterior precision is diagonal with entries
$$\lambda_i\gamma_j + 1,$$
where $\lambda_i$ and $\gamma_j$ are the rater and shape eigenvalues. Exact samples, exact log-determinants and exact marginal variances therefore cost $O(R^3 + K^3)$, not $O(R^3K^3)$.
Topology is guaranteed by the representation itself: every sample is a positive radial function and therefore one connected region without holes. The anatomical prior shapes which region is likely. The rater covariance ties the rows together, and that coupling is what lets the model borrow strength across raters and fill in missing ones.
The novelty is not "use a Sylvester solver." It is the observation that multi-rater contour modelling has exactly two natural axes, raters and anatomy. A Kronecker-structured Gaussian over those two axes makes consensus estimation, uncertainty sampling and rater imputation all exact linear algebra.
Relation to Prior Work
Three works motivate the gap. Samei et al. fit a PCA shape model of 78 MRI prostates to a V-Net probability map using particle swarm optimisation, a derivative-free search. They report a mean Jaccard of 90.58% on 3D ultrasound. Their fit produces one consensus contour and cannot pass gradients back to the network. We keep their shape model but make the fit closed-form and differentiable, and we add the rater axis. Liu et al. encode rater style in a 6-dimensional latent space of a variational autoencoder (a generative model trained to reconstruct its input through a compressed random code) and decode samples with a full network. Pionono gives each rater its own Gaussian in latent space, with the mean capturing bias and the variance capturing inconsistency. Both model raters well. Neither constrains the anatomy of what they decode, and both need repeated decoder passes for a distribution. We move the rater distribution from a latent space into outline space, where the anatomical prior applies directly and sampling needs no decoder. DiFF shows that majority-vote consensus throws away clinically useful borderline cues, with glaucoma AUC gains of up to 2.73% from diagnosis-aware fusion. It smooths rater weights with a heuristic convolutional module. It constrains our evaluation: we must show that anatomical regularisation does not erase the extremes that matter downstream.
Two works supply the method. Kao & Hennequin show that the backward pass of a Sylvester solve is itself a Sylvester solve, so the layer can be trained without unrolling iterations. We use this directly. Wang et al. couple a deep prior image with a physical model through a Sylvester equation in hyperspectral super-resolution and tune the prior weight by a one-dimensional search. Their design is the closest in form, a network prior combined with a closed-form quadratic solve. It has a single ground truth and no uncertainty, and our learned $D$ and $\Sigma_r$ replace their scalar weight. The Semantic Autoencoder is the canonical demonstration that a two-sided quadratic objective collapses to a Sylvester equation independent of sample count. Our computational argument rests on it.
Research Program
The network is a standard 2D or 3D U-Net encoder. It has a small head that predicts the structure's centre and scale, plus a rater-conditioned radial head that outputs $F$ and $D$. In 2D, rays are sampled at $K = 128$ angles. In 3D, they are sampled at $K \approx 1000$–$2500$ quasi-uniform sphere directions. $\Sigma_s$ is initialised from PCA on all training outlines of all raters, expressed in log-radius after centring. It is then refined as a low-rank-plus-diagonal matrix, with a small smoothness kernel added so that residual variation along the boundary is smooth. $\Sigma_r$ is low-rank plus diagonal over the $R$ raters.
Training maximises the exact Gaussian log-likelihood of the observed rater outlines under the posterior predictive, computed in the joint eigenbasis. Gradients flow through the Sylvester solve via the adjoint rule. When a training image lacks some raters, the Kronecker structure breaks. In that case we solve the conditioned system by conjugate gradients preconditioned with the full-panel Sylvester solve, and we report how often this path is needed.
For optic disc and cup, nesting is enforced by modelling disc log-radius and the logit of the cup-to-disc radius ratio as two stacked shape blocks. Every sampled cup then lies inside its disc.
At test time there are three outputs. The mean solution gives per-rater contours. Posterior samples give a distribution, which is converted to the per-pixel frequency maps that QUBIQ's staged Dice expects. Conditioning on any subset of rater rows gives imputed contours for the others.
The analysis side asks what $\Sigma_r$ learns. We check whether its leading eigenvector matches a global "generous versus conservative" axis, and whether rater clusters agree with known seniority where the metadata allows. We also ask where along the boundary the posterior variance concentrates, for example whether it peaks at the prostate base and apex, where Samei et al. report the lowest agreement.
Baselines
The naive reference is a single U-Net trained on the soft average of rater masks. It produces one probability map, and QUBIQ's thresholding turns that map into a pseudo-distribution. It costs one forward pass. This shows how much of the staged Dice is available without modelling raters at all.
The strongest existing competitor is a deep ensemble of deterministic U-Nets. The QUBIQ challenge synthesis reports that the top ensemble entries reached average staged Dice between 0.80 and 0.85 across tasks, beating every single-model probabilistic method. We run five members on the same backbone, the same augmentations and the same epoch budget per member, so the ensemble receives five times our training compute. We report both matched-compute and ensemble-favoured comparisons. Beating it means matching its staged Dice within the seed-to-seed spread on brain growth, prostate and kidney, while having zero topology violations, one forward pass, and rater-specific and imputed outputs that an ensemble cannot produce.
The strongest rater-aware competitors are the variational model of Liu et al. and Pionono. Liu et al. report a continuous Dice of 0.5372 on QUBIQ brain growth, against 0.5083 for Bayes dropout and 0.5042 for ESULR. That is the number to beat on that task under their protocol, and we will reproduce their protocol before reporting ours. Pionono is reported only on histology datasets in the file: κ = 0.758 on Gleason 2019 and κ = 0.711 on the 25-rater breast dataset. Neither dataset involves single-component outlines, so we re-implement it on our backbone and tasks with the same tuning budget of 20 configurations. No reported result was found on Lacuna for Pionono on QUBIQ.
On REFUGE-2, the downstream competitor is DiFF. The file gives only relative AUC gains of up to 2.73%, not absolute values, so we re-run it. We compare glaucoma AUC computed from each method's vertical cup-to-disc ratio.
Two ablations isolate the key ingredient. The first sets $\Sigma_r = I$. The Sylvester equation then decouples into independent ridge projections of each rater's outline onto the shape model, which is simple closed-form PCA regularisation. The second removes the shape prior by setting $\Sigma_s$ to a scaled identity, which keeps the rater coupling but drops the anatomy. The proposal is justified only if the full model beats both ablations on imputation and calibration.
Why not just…?
Why not just use a deep ensemble and post-process each member for topology, keeping the largest connected component and filling holes? This is the most serious threat, because ensembles already lead QUBIQ. Post-processing fixes islands and holes but not implausible shapes, and it gives no rater identities and no way to condition on observed raters. We will run exactly this "ensemble + cleanup" baseline. If it matches our staged Dice and calibration at comparable latency on 2D tasks, we will concede the 2D case. We would then rest the claim on 3D cost and on rater imputation, which the ensemble cannot do.
Why not just project each rater head's output onto the PCA shape model independently? That is the $\Sigma_r = I$ ablation, and it is also closed-form. If it performs as well, the rater coupling, and with it the Sylvester structure, adds nothing and we will say so. The prediction that separates the two is held-out-rater imputation. With independent rows, observing two raters says nothing about the third beyond the image. With coupled rows, it should cut the imputation error substantially.
Why not just solve the vectorised linear system, or run gradient descent or the alternating direction method of multipliers (ADMM)? In 2D with $R = 7$ and $K = 128$, the vectorised system has 896 unknowns and a dense solve is cheap, so the Sylvester form is not needed for speed there. We will say so. The structure matters in two places. The first is exact sampling and log-determinants, which require the joint eigenbasis. The second is 3D, where $R = 6$–$25$ and $K \approx 2500$ put the vectorised system in the tens of thousands of unknowns. We will report wall-clock time and memory for the Sylvester, dense vectorised and unrolled-gradient solves at matched accuracy. This follows the order-of-magnitude gaps reported by the Semantic Autoencoder and the iteration savings in Kao & Hennequin, rather than assuming them.
Why not just use Monte Carlo dropout? On QUBIQ brain growth it already trails structured latent models, at 0.5083 versus 0.5372 in Liu et al.. It also needs many forward passes, so we include it only as a secondary reference.
Proposed Experiments
We evaluate on the QUBIQ tasks with single-component targets: brain growth MRI (7 raters), prostate MRI and kidney CT. Kidney is run in 3D with spherical rays. We add REFUGE-2 optic disc and cup, with vertical cup-to-disc ratio feeding a fixed glaucoma classifier. Pancreas, pancreatic lesion and brain tumour are reported as explicit failure cases of the radial representation, not tuned for.
The metrics are as follows. Distributional fit is measured by QUBIQ staged Dice. Rater-specific fidelity is measured by Dice and boundary distance to each rater's own outline when the rater is known. Calibration is measured by the correlation between posterior boundary variance and the empirical spread across raters along each ray. Topology violation rate is the fraction of samples with more than one component or with holes. Cost is measured as latency and peak memory per volume. On REFUGE-2 we also measure glaucoma AUC.
The first decisive check is held-out-rater imputation. On test images, we reveal one or two raters' outlines and score the predicted outlines of the rest against the truth. We compare the full model, the $\Sigma_r = I$ ablation, and a Pionono variant with its rater posteriors conditioned where feasible.
The second decisive check tests whether anatomy erases clinically meaningful extremes. On REFUGE-2, we compare the widest and narrowest sampled cup-to-disc ratios against the widest and narrowest rater ratios, and we check that glaucoma AUC is not below DiFF's. If shrinkage toward the mean shape suppresses the extreme raters, it will show up here. The remedy would then be heavier-tailed rater factors rather than a stronger prior.
The third decisive check is scaling. On 3D kidney, we compare Sylvester solve cost against the dense vectorised solve and against a five-member 3D ensemble as $K$ grows from 500 to 2500. The additive-cost claim predicts roughly cubic growth in $K$ alone, independent of $R$.
References
- Automatic Segmentation of the Prostate on 3D Trans-rectal Ultrasound Images using Statistical Shape Models and Convolutional Neural Networks
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Automatic differentiation of Sylvester, Lyapunov, and algebraic Riccati equations
- Hyperspectral Image Super-Resolution via Deep Prior Regularization With Parameter Estimation
- Semantic Autoencoder for Zero-Shot Learning
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge