When two radiologists outline the same tumor, they rarely agree. Much of the disagreement is systematic rather than random: one draws consistently wider margins, another excludes an ambiguous edge that a third includes. A recent line of work learns these habits as a small "style" vector per annotator, so that one network can draw a scan the way a given expert would. These models only work offline, though. They need the rater's identity at training time, and they give a new clinician no way to steer the output. Today's interactive tools have the reverse problem. They respond to clicks, but they treat every correction as a purely local fix toward a single "true" mask, so a clinician who widens the margin on slice 40 must widen it again on slices 41 through 80.
We propose an editor that treats each correction as evidence about who the clinician is. It runs at two speeds, borrowing the design of two-speed ("multi-rate") robot controllers, where a slow planner sets goals and a fast reflex loop executes them. The fast loop makes the stroke under the clinician's pen take effect immediately on the slice being edited. The slow loop, running about once a second over the whole volume, updates a belief about the clinician's style vector from all strokes so far and redraws every untouched slice in that inferred style. The research question is how to divide each correction into the part that reflects a habit, which should propagate across the volume, and the part that is a one-off local fix, which should stay put. The claim that can be tested is simple: a clinician should reach their own contour in fewer strokes than with local-only editing, without the untouched slices getting worse.
Core Claim
Multi-rater latent models already separate systematic bias from inconsistency. Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono) models each rater as a Gaussian in an 8-dimensional latent space. The mean of that Gaussian encodes the rater's consistent bias and its spread encodes how inconsistent they are with themselves. We reuse this split as the attribution rule for interactive edits. The part of a stroke that a shift in the style vector can explain is treated as habit and propagated everywhere. The remainder is treated as a local event and handled by a fast residual correction confined to the stroke's neighborhood.
Write $D(x, z)$ for the multi-rater decoder applied to image volume $x$ with style vector $z$. Write $s_1,\dots,s_n$ for the clinician's strokes, each a set of boundary points marked inside or outside. The slow loop maintains
$$ p(z \mid s_{1:n}) \;\propto\; p(z)\prod_{i=1}^{n} p\big(s_i \mid D(x,z)\big), $$
where $p(z)$ is the population prior over rater styles learned from the training raters. The likelihood scores how well stroke $s_i$ agrees with the boundary that style $z$ predicts. The fast loop then applies a residual displacement $r_i$, which is the part of $s_i$ that the posterior-mean decoding does not explain. It decays with distance from the stroke, at a rate set by the posterior spread: the more inconsistent this clinician appears, the more local the residual stays.
This design is new in a specific sense. It does not simply combine an interactive segmenter with a probabilistic one. It uses the inter-observer model as a prior that turns sparse corrections on a few slices into a whole-volume, personalized contour.
Research Program
The base model is a multi-rater segmenter with an explicit low-dimensional style latent. We start from Pionono's per-rater Gaussians and, as a second backbone, the bound-constrained latent space of Diversified and Personalized Multi-Rater Medical Image Segmentation (D-Persona). D-Persona learns a shared latent space constrained to predict the intersection and union of all raters' masks, then projects individual rater identities into it. Both papers assume the rater is known at test time. We replace the identity lookup with posterior inference from strokes.
The two-speed structure follows lessons from robotics, and those lessons constrain the design as much as they supply it. Compliant Residual DAgger couples a slow frozen base policy at 1 Hz with a roughly 2 MB residual policy at 50 Hz trained on human corrections. It found that letting the human add small corrections, rather than taking over control, avoids jerky discontinuities. That paper also shows the residual only helps when the base is already partly correct, which is why we start from a population-prior decoding rather than from scratch. EvoControl splits control into a slow planner at 31.25 Hz and a fast reflex loop at 500 Hz. It warns that the fast loop chases a moving target when the slow loop changes underneath it, and it handles this by gradually shifting from a fixed analytical (PD) controller to a learned one.
Our version of that moving-target problem is concrete. When the style belief updates, residuals stored as mask differences become wrong. We therefore store strokes as anchored constraints, not as pixel changes, and recompute residuals against each new slow-loop decoding. In the same spirit, the first fast loop is a fixed geometric deformation around the stroke. A small learned residual network is added only if it beats the fixed deformation.
The slow loop's work is posterior inference over 6–8 dimensions plus one volumetric decode. It runs asynchronously and may take up to a second or two. The fast loop touches only the current slice and must keep pace with pen motion, with a target of 30–60 Hz. These rates are design targets, not measured results.
We also build a stroke simulator for evaluation. For a held-out rater, it repeatedly picks the largest boundary disagreement between the current output and that rater's mask, on the slice a user would plausibly inspect next, and draws a stroke along it with added positional noise.
A tension raised by Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle (DiFF) shapes one reported diagnostic but stays outside the method. DiFF weights raters by how much their contours help a downstream diagnosis classifier and argues that some rater variation is clinically harmful. Faithfully propagating a clinician's style could therefore propagate a harmful habit. We do not build a utility-based outer loop here, since that would be a second project. We do report whether inferred styles fall outside the range of the training raters.
Baselines
The naive reference is local-only editing. A consensus-trained model makes the initial prediction, and each stroke changes only its own neighborhood through the same fast-loop deformation our system uses. Every stroke-count gain must be measured against this. Because the fast loop is identical, any difference is attributable to style propagation.
The strongest non-interactive competitor is D-Persona when told the rater's identity. It reports a personalized mean Dice of 76.19% and a Generalized Energy Distance of 0.2359 on NPC-170, and a personalized mean Dice of 89.17% on LIDC-IDRI (D-Persona). Knowing who the rater is makes this an oracle for the style component. We retrain it on our splits with the same backbone and tuning budget. We count our method as beating it if, starting from an unknown clinician, it matches or exceeds that rater-known Dice within a small fixed number of strokes, which we fix in advance at five. D-Persona's own authors note that personalized models still fall short of single-expert specialists, the "expert ceiling". A decisive win would be exceeding the oracle, because local residuals capture what the latent cannot.
For the interactive baseline, we use the click-based iterative refiner of Iteratively-Refined Interactive 3D Medical Image Segmentation With Multi-Agent Reinforcement Learning (IteR-MRL). It refines a voxel probability map from clicks and reports Dice 88.53 on BraTS2015 within 5–10 clicks. We also use SegVol, a promptable 3D foundation model reporting 0.8593 average Dice on AMOS22 and about 17x speedup over sliding-window inference. Neither reports results on NPC-170 or LIDC-IDRI against individual raters: no reported result found on Lacuna for interactive per-rater segmentation on these datasets. We rerun both with our stroke simulator, converting each stroke into the clicks they accept, and target the held-out rater's mask.
Three ablations isolate the key ingredient. The first is the slow loop only: posterior style inference with no residual, which tests whether a 6–8-dimensional latent can express the corrections at all. The second is a synchronous variant that runs full posterior update and decode on every stroke, which tests whether the two-speed split matters or is only engineering. The third applies uniform attribution, propagating every correction with fixed stiffness and ignoring the inferred inconsistency, which tests whether the inter- versus intra-observer split earns its place.
Why not just…?
Why not just fit the style vector to the strokes and skip the fast loop? This is the most serious alternative, and it is the slow-loop-only ablation. It will likely capture global margin habits well. A low-dimensional latent, however, cannot represent a correction that is specific to one lesion lobe, and Pionono's modest agreement figures (unweighted κ of 0.758 on Gleason 2019 and 0.711 on the 25-rater breast set) suggest much of each rater's contour is not explained by the latent. If the slow-loop-only variant matches the full system on strokes-to-target, the residual loop is unnecessary and we will say so.
Why not just use a fast interactive model like SegVol and let the clinician correct each slice? SegVol is fast, but it produces one deterministic mask per prompt and has no notion of whose contour it is drawing. Each correction therefore stays where it is made. The local-only baseline and the rerun of SegVol measure exactly this. The proposal fails if their stroke counts are not meaningfully higher than ours on volumes with many slices.
Why not just run everything synchronously, since a single decode may already be fast enough? It may be. The robotics evidence (EvoControl; Deep Neural Networks for Improved, Impromptu Trajectory Tracking of Quadrotors, where a 7 Hz reference generator outside a 70 Hz loop cut tracking error by 43%) concerns physical instability, which does not exist in a contour editor. The case for two speeds here rests only on responsiveness while drawing. The synchronous ablation, timed on the same hardware, decides whether the architecture is necessary or merely convenient. If synchronous latency stays within interactive bounds on NPC-170 volumes, we report the split as optional.
Why not just precompute an ensemble of rater styles and let the clinician pick one? Deep ensembles topped QUBIQ, where the best teams scored average Dice of 0.80–0.85, but they are costly in 3D. A menu of fixed styles also cannot represent a new clinician who falls between the training raters. We include a "pick the best training rater" baseline. Beating it requires that posterior inference land on styles that no single training rater matches.
Proposed Experiments
The main benchmarks are NPC-170, 3D MRI with four experts per scan, and LIDC-IDRI, thoracic CT with multiple radiologists, because only 3D volumes let style propagate across slices. We add the 25-rater breast cancer histopathology set used by Pionono. It is 2D, but its many raters allow genuine leave-one-rater-out evaluation, with propagation across regions of a slide instead of across slices. QUBIQ brain growth, with seven raters, serves as a small sanity check. On it, Variational Inference for Quantifying Inter-observer Variability reports a continuous Dice of 0.5372.
In every experiment the target rater's annotations are withheld from training wherever rater identities permit, so the system faces a clinician whose style it has never seen.
The primary metric is Dice and average surface distance to the held-out rater's mask as a function of stroke count, summarized as strokes needed to reach a fixed Dice threshold. Two secondary metrics follow. Untouched-slice drift measures the change in Dice on slices never edited, which should rise for us and must not fall. Latency is measured for each loop.
Three checks are decisive. The first tests whether the full system needs fewer strokes than local-only editing on 3D volumes, with the gap growing with the number of slices. The second tests whether it reaches D-Persona's rater-known Dice within five strokes on NPC-170 and LIDC-IDRI. The third tests whether the calibrated attribution beats uniform attribution specifically on raters whose held-out annotations are internally inconsistent. If the second check fails and the slow-loop-only ablation matches the full system, the honest conclusion is that stroke-driven style inference is the contribution and the two-speed architecture is not.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Diversified and personalized multi-rater medical image segmentation
- Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections
- EvoControl: Multi-Frequency Bi-Level Control for High-Frequency Continuous Control
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Iteratively-Refined Interactive 3D Medical Image Segmentation With Multi-Agent Reinforcement Learning
- SegVol: Universal and Interactive Volumetric Medical Image Segmentation
- Deep neural networks for improved, impromptu trajectory tracking of quadrotors
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Variational inference for quantifying inter-observer variability in segmentation of anatomical structures