When several radiologists outline the same tumor, they rarely agree. Current methods summarize that disagreement as a single number per pixel, such as a variance or a "fraction of raters who included this pixel." A pixel with value 0.5 means half the raters included it. That number cannot say whether those raters drew a larger version of the same shape, or whether they placed the boundary somewhere else entirely, shifted sideways along the organ. This proposal treats inter-observer disagreement as a direction field instead. At every point on the boundary, the field records which way the contour moves as one goes from cautious raters to generous raters, and how much each rater additionally slides the boundary sideways. We learn that field from the image and render it with continuous flow encodings borrowed from traffic and mobility visualization, such as tapered ribbons and hue-coded orientation, rather than arrows.
The first question is empirical, and it determines whether the idea matters at all. How much of real multi-rater disagreement is "nested," meaning one rater's mask contains another's, and how much is "crossing," meaning the boundaries cross each other? A per-pixel probability map captures nested disagreement exactly, because thresholding it at different levels produces a nested family of contours. It loses only the crossing part. If crossing disagreement is a substantial share of the disputed area on public multi-rater benchmarks, scalar maps systematically misrepresent what experts disagree about, and a two-part model (a shared growth direction plus a per-rater sideways displacement) should predict individual raters' contours better. If crossing disagreement is negligible, the proposal collapses to a visualization of the soft map, and we will say so.
Core Claim
Multi-rater boundaries can be written as a shared scalar "potential" whose level sets give the nested part, plus a small per-rater tangential displacement field that gives the crossing part. Concretely, rater $r$'s mask is modeled as
$$M_r = W_{v_r}\big(\{x : \phi(x) < t_r\}\big),$$
where $\phi(x)$ is an image-conditioned potential predicted by a segmentation network, $t_r$ is a scalar for rater $r$ that places the rater on a cautious-to-generous ordering, $v_r(x)$ is a smooth displacement field restricted to point along the boundary rather than across it, and $W_{v}$ warps a mask by $v$. As $t$ increases, the boundary flows along $-\nabla\phi$. This is the "expansion flow," and its ordering is directional rather than symmetric: rater A subsuming rater B is not the same relation as B subsuming A. The fields $v_r$ carry everything a probability map cannot. The claim is that $v_r$ is non-negligible on real data, that it is predictable from the image and a few examples of the rater, and that showing it to clinicians changes how well they anticipate where a colleague will draw.
Relation to Prior Work
Three lines of work model rater variability, and all of them output scalar or mask-level summaries. Pionono, from Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation, gives each annotator a Gaussian in an 8-dimensional latent space and decodes per-rater masks. The variational model in Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures samples a 6-dimensional latent to produce diverse boundary hypotheses. The coupled networks in Disentangling Human Error from Ground Truth in Segmentation of Medical Images learn pixel-wise rater confusion matrices around a single hidden truth. These papers supply the problem setting and the strongest competitors. In principle Pionono can already generate per-rater masks, so our contribution is not "per-rater prediction." It is a factorization that separates growth from sideways shift and makes the direction of disagreement an explicit, inspectable quantity. Pionono's symmetric latent geometry has no notion of one rater subsuming another.
Two further papers motivate keeping raters separate rather than averaging them. More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel shows that keeping one model per pathologist beats training on consensus in both F1 and calibration. QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge defines the benchmark we use, but it scores predictions only through thresholded soft maps, which by construction reward only the nested part.
Two papers constrain the design. From Registration Uncertainty to Segmentation Uncertainty found that raw deformation magnitude correlates almost not at all with label error (r < 0.10). We therefore do not use unconstrained registration between rater masks as our displacement. Instead we fit $v_r$ only on the boundary, only in the tangential direction, and only after the nested part has been removed. Beyond scalar losses: calibrating segmentation models via gradient vector field surgery uses vector fields to calibrate boundaries, but only internally during training and only against single consensus masks. We borrow its observation that Dice-style losses produce overconfident margins, and train $\phi$ with a calibrated loss.
The visualization side supplies the rendering method. Tapered segments in Visualizing Statistical Mix Effects and Simpson's Paradox and Topology Density Map for Urban Data Visualization and Analysis convey direction without arrowheads. Hue-coded orientation, as in A Method for Visualizing Moving Objects from Aerial Images for Monitoring Traffic Conditions, encodes angle continuously. Clutter in dense arrow plots is documented in Interactive visual exploration and analysis of origin-destination data. We render the expansion flow as ribbons that taper from the most cautious contour to the most generous, and render $v_r$ as hue along the boundary.
We considered applying Finsler Multi-Dimensional Scaling: Manifold Learning for Asymmetric Dimensionality Reduction and Embedding to embed raters asymmetrically. Our scalar ordering $t_r$ is the one-dimensional special case of its drift idea. With 4 to 7 raters per dataset, a full asymmetric embedding would be fit to too few points to be meaningful, so we keep the one-dimensional version.
Research Program
The work has three stages. The first stage is measurement and needs no learning. For every image in the multi-rater datasets, we compute the largest nested sub-family of rater masks and the "crossing area": the disputed area that no single rater ordering can explain. We report this as a fraction of total disputed area, per task and per structure. This number decides whether the rest of the program is about modeling or merely about display.
The second stage is the model. A ResNet34 U-Net, the same backbone Pionono uses, predicts $\phi$. A per-rater embedding predicts $t_r$ and the coefficients of $v_r$ in a low-dimensional basis of smooth tangential fields along the predicted boundary. The model is trained to reconstruct each rater's mask. A smoothness penalty and a magnitude penalty on $v_r$ ensure the crossing part is used only when the nested part cannot explain the data.
The third stage is a reader study. Radiologists see either a soft probability map, sampled Pionono contours, an arrow-glyph rendering, or the flow rendering. Their task is to predict where a specific held-out colleague will draw, and to classify a disputed region as "same shape, different size" or "different placement."
Baselines
The naive reference point is a single consensus mask produced by STAPLE or majority vote, used as a deterministic prediction for every rater. It is what clinical pipelines do. Grove et al. report that consensus training costs both F1 (72.9% versus 78.2% for the full panel on DEPP) and calibration (ECE 0.183 versus 0.050).
The natural scalar competitor is the mean soft map trained with a calibrated loss and thresholded at each rater's best-fitting level. This model is exactly our nested-only ablation, and it isolates the value of $v_r$. It gets the same backbone, data, and hyperparameter search budget as the full model.
The strongest existing method is Pionono. On Gleason 2019 it reports an unweighted Cohen's kappa of 0.758, and on the 25-rater breast cancer set a kappa of 0.711. On QUBIQ brain growth, the strongest number in the file is a continuous Dice of 0.5372 from the variational model of Liu et al. For LIDC-IDRI, no reported result was found on Lacuna for any of these methods under a per-rater metric.
We rerun Pionono and the variational model with their published latent sizes on our splits, using the same backbone and the same number of tuning trials. Our model beats them only if it reaches higher per-rater Dice on held-out images and lower boundary distance for held-out raters given five example annotations, at equal or better calibration. Matching overall soft-map Q-Dice alone does not count.
A second ablation replaces the tangential, boundary-restricted $v_r$ with an unconstrained per-rater registration warp. This tests whether the restriction motivated by the registration-uncertainty paper actually matters.
Why not just…?
Why not just threshold a calibrated soft probability map? Thresholding a soft map at different levels reproduces every nested disagreement. If the stage-one measurement finds that crossing area is a small share of disputed area, this alternative wins and the modeling contribution is moot. That is a real threat, and stage one exists to settle it before any model is trained. The nested-only ablation then quantifies the remaining gap on held-out raters.
Why not just sample per-rater masks from Pionono and overlay the contours? Pionono can generate rater-specific contours, but its symmetric latent space does not separate "this rater draws bigger" from "this rater draws elsewhere." Overlaying four to seven contours brings back the clutter problem documented for dense directed lines. Whether the explicit factorization buys anything beyond a re-rendering of Pionono samples is tested twice. Computationally, the held-out-rater comparison asks which model predicts an unseen rater better. Behaviorally, the reader study compares the overlaid-sample condition against the flow rendering.
Why not just draw arrows along the boundary normals? Arrow glyphs are the simplest direction encoding. The origin-destination and social-network visualization papers report that they become unreadable at density, and a boundary sampled every few pixels is dense. Arrows are included as an explicit condition in the reader study rather than dismissed, because the clinical boundary setting is sparser than city-scale traffic and might tolerate them.
Proposed Experiments
The data are the six QUBIQ tasks (prostate, brain growth, brain tumor, kidney, pancreas, pancreatic lesion), LIDC-IDRI with four radiologists, and the 25-rater breast cancer set used by Pionono, which tests whether $t_r$ orderings stay stable when there are many raters. The metrics are QUBIQ's Q-Dice over thresholds 0.1 to 0.9 for comparability, per-rater Dice and symmetric boundary distance on held-out images, few-shot held-out-rater boundary distance, and ECE.
The first decisive check is stage one. A crossing-area fraction below roughly one tenth on most tasks would reduce the project to a visualization study. The second is whether the full model beats the nested-only ablation on held-out-rater prediction by a margin larger than seed variance across five seeds. The third is whether $v_r$ is reproducible: fitted to two disjoint halves of a rater's cases, its direction along matched boundaries should agree better than for shuffled rater identities. If it does not, the "sideways style" is noise. The fourth is the reader study. Radiologists using the flow rendering should predict a held-out colleague's boundary more accurately, and classify disputed regions as "size" versus "placement" more accurately, than with the soft map, overlaid samples, or arrows. Accuracy is scored against that colleague's actual annotation.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures
- Disentangling Human Error from Ground Truth in Segmentation of Medical Images
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- From Registration Uncertainty to Segmentation Uncertainty
- Beyond scalar losses: calibrating segmentation models via gradient vector field surgery
- Visualizing Statistical Mix Effects and Simpson's Paradox
- Topology Density Map for Urban Data Visualization and Analysis
- A Method for Visualizing Moving Objects from Aerial Images for Monitoring Traffic Conditions
- Interactive visual exploration and analysis of origin-destination data
- Finsler Multi-Dimensional Scaling: Manifold Learning for Asymmetric Dimensionality Reduction and Embedding