Additive Disagreement Modules: Muscle-Synergy Factorization for Annotator Style in Medical Segmentation

When several doctors outline the same tumor or organ, their outlines differ. Some of that difference comes from the image itself, such as a blurry edge that nobody can place exactly. Some comes from the person: one radiologist reliably draws a few millimetres wide into low-contrast tissue, while another reliably stops short. Current models of this "inter-observer variability" give each annotator a point in an abstract Gaussian latent space and decode a mask from it. Those latent axes carry no meaning, and they can be negative or mix freely. So the models can imitate a doctor without saying how that doctor differs from colleagues, and they offer no principled way to describe a new doctor from a handful of examples.

Motor neuroscience faced a structurally similar problem and solved it with non-negative matrix factorization (NMF), a method that writes a non-negative data matrix as a product of two non-negative matrices, so every observation is an additive mix of a few parts. Applied to muscle activity, NMF yields "synergies": a small set of muscle-group patterns plus per-trial recruitment weights. These patterns turn out to be conserved across conditions and to reorganize in interpretable ways after injury. We propose the same treatment for annotator disagreement. The core claim is that, once disagreement is measured in local image coordinates and split into over-contouring and under-contouring parts, it factorizes into a few additive "disagreement modules", such as "expands into low-gradient tissue" or "cuts high-curvature corners". Each annotator then becomes a short non-negative recruitment vector over those modules. That vector should be stable within an annotator, estimable from five to ten examples, and usable to generate that annotator's masks at almost no extra inference cost.

What is new, and what is not

The algebra of NMF on masks is not the contribution. The contribution has three parts. The first is a choice of coordinates in which disagreement is additive and comparable across images. The second is a set of validation methods borrowed from synergy research: cross-condition conservation, rank selection, and a test of real structure versus artifact. The third is a direct test of whether annotator style has more than one dimension at all.

The closest prior work is Pionono, Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation. It gives each rater a Gaussian in an 8-dimensional latent space attached to a U-Net. The Gaussian's mean encodes the rater's systematic bias and its variance encodes the rater's self-inconsistency. We keep that mean/spread split, but place it in non-negative, nameable coordinates rather than an unconstrained latent space.

Two other models are related. Variational Inference for Quantifying Inter-observer Variability in Segmentation of Anatomical Structures samples a 6-dimensional standard-normal latent to produce plausible masks, but it has no rater identity at all. The Latent Doctor Model learns a doctor embedding from a support set of that doctor's labels. That is the few-shot idea we also pursue, but it does so only for grading and classification, not dense masks.

Multi-decoder architectures spend a full decoder per rater. These include Multi-decoder U-Net and OMBA-UNet (Rater-Specific Bayesian Neural Networks); the latter reports 0.34 s per slice against 0.02 s for a plain U-Net. Our generator replaces those decoders with K additive maps.

DiFF learns weights for blending rater masks to maximize downstream diagnosis. It motivates the view that rater differences matter, but it blends whole masks rather than decomposing them into parts.

On the motor side, the modularity editorial and the locomotion work in Effects of Perturbations to Balance supply the method and its rationale: non-negativity matches quantities that only add. Two further studies supply the main validation idea, that modules should be conserved while recruitment changes. Oliveira et al. found conserved muscle-weighting patterns across treadmill and overground running. Sit-to-Walk found that a transition movement reuses existing modules. The stroke synergy studies supply a vocabulary for comparing groups: modules can be preserved, merged, shared, or lost.

Could a Neuroscientist Understand a Microprocessor? constrains the design. It shows that dimensionality reduction yields tidy components with no causal meaning. We therefore build in planted-structure recovery and a test that separates image-driven structure from rater-driven structure, rather than trusting neat-looking modules.

Research Program

Disagreement coordinates. For rater $r$ on image $i$, we take as reference the mean mask of the other raters (leave-one-rater-out). We split rater $r$'s deviation from that reference into two non-negative maps: pixels the rater adds, and pixels the rater removes.

Raw pixel positions are not comparable across images. We therefore bin pixels by a small set of local descriptors:

  • signed distance to the reference boundary,
  • local intensity-gradient magnitude,
  • boundary curvature,
  • for histopathology, the tissue class of the reference.

Within each of $B$ bins we record the excess-inclusion rate and the excess-exclusion rate. This gives each rater–image pair a non-negative vector of length $2B$. For the multi-class Gleason labels we repeat the construction per class in a one-versus-rest fashion.

Factorization. Stacking these vectors gives

$$V \approx W H, \qquad V \in \mathbb{R}_{+}^{2B \times N},\; W \in \mathbb{R}_{+}^{2B \times K},\; H \in \mathbb{R}_{+}^{K \times N},$$

where $N$ is the number of rater–image pairs and $K$ is the number of modules. Each column of $W$ is a disagreement module, and each column of $H$ gives how strongly one rater recruited each module on one image. A rater's style is the average of their columns of $H$. Their self-inconsistency is the spread of those columns across images, in the same sense as Pionono's mean and variance.

Synergy work picks $K$ by a variance-accounted-for threshold, a heuristic the file flags as an open problem. We instead choose $K$ by how well the model reconstructs disagreement for held-out images. That choice is a small methodological contribution back to the synergy side.

Generator. A single segmentation network predicts two things: a soft consensus map $p_0(x)$, and per-pixel descriptor-bin memberships. Through $W$, those memberships give each module $k$ a spatial expansion map $a_k(x)$ and a retraction map $b_k(x)$. Rater $r$'s predicted mask is then

$$p_0(x) + \sum_k h_{rk}\,\big(a_k(x) - b_k(x)\big),$$

clipped to $[0,1]$. Here $h_{rk}\ge 0$ is the rater's recruitment of module $k$. A new rater is calibrated by fitting $h_r$ with non-negative least squares from $n$ annotated images, with no gradient steps.

The rank-one question. If annotator style were only "draws bigger or smaller", one module would suffice. We test this explicitly. A finding that $K=1$ is enough would itself be a useful negative result.

Baselines

The naive reference is a U-Net trained on majority-vote (or STAPLE) labels, which predicts the same mask for every rater. Prior work shows this is weak. The Latent Doctor Model reports that majority-vote training reproduces the rater distribution worse than distribution-preserving models, whose CLDM variant reaches a KL divergence of 0.461 on Gleason 2019. Any rater-aware method must clear this baseline clearly.

The most dangerous simple baseline is per-rater thresholding. It trains the same consensus network and gives each rater one scalar: the threshold on $p_0$ that best matches their training masks. This captures pure dilation/erosion style with one parameter and costs nothing at inference. It is exactly the $K=1$ special case of our model, so beating it is the test of whether multiple modules are real. We count it beaten only if the full model improves per-rater held-out kappa or Dice beyond seed-to-seed variation across five seeds.

The non-negativity ablation runs PCA, or unconstrained low-rank factorization, on the same signed descriptor vectors, with the same $K$ and the same generator and calibration. It isolates whether additivity matters. We also ablate the coordinates by running NMF on raw resized pixel deviation maps, which isolates the descriptor binning.

The strongest existing method is Pionono. The file reports unweighted Cohen's kappa of 0.758 for it on Gleason 2019 (against 0.750 for the previous best) and 0.711 on the 25-rater bc dataset. We rerun it with the same ResNet34 encoder, training split, and hyperparameter-search budget. For the few-shot experiment we give it the same $n$ images per new rater and fit its latent mean by gradient descent on those images. Beating it means matching its kappa within noise while winning on few-shot new-rater accuracy, module stability, or latency. We do not claim to beat it on raw kappa.

On QUBIQ we compare against the multi-decoder U-Net. It reports 0.852 and 0.871 on the two brain-tumor tasks, 0.870 on kidney, and 0.553 on prostate task 1 under the challenge metric, and top QUBIQ teams averaged 0.80 to 0.85 Dice. We also compare against a per-rater deep ensemble at matched training compute, and report inference latency next to OMBA-UNet's 0.34 s/slice. For GED on LIDC-IDRI we found no reported result on Lacuna, so we report our own reruns only.

Why not just…?

Why not just give each rater a threshold on a consensus map? This is the real threat, because many raters may differ mainly in overall generosity. We do not argue around it. It is the $K=1$ case of our model, and the held-out rank-selection curve directly shows whether more modules help. If the curve is flat at $K=1$ on Gleason 2019 and bc, the claim fails, and we would report that annotator style there is essentially one-dimensional.

Why not just use PCA on the disagreement vectors? PCA allows negative recruitment. That means a rater could "un-expand into low-contrast tissue", and a component could cancel against another, which makes the parts unnamable. The ablation keeps everything else fixed. If PCA matches NMF on reconstruction and on cross-split module stability, the non-negativity argument from the motor literature does not transfer, and we would say so.

Why not just use Pionono or a deep ensemble? Both predict well. The file notes that the Latent Doctor authors consider per-doctor ensembles very strong, and that QUBIQ ensembles often beat probabilistic models. Neither gives named, transferable components, though. Neither calibrates a new rater from a handful of images without retraining, and ensembles scale linearly in compute. The few-shot curve and the latency measurement settle whether those advantages are real or cosmetic.

Proposed Experiments

The main rater-identity experiments use Gleason 2019 (6 pathologists, 244 cores) and bc (25 raters). The bc dataset allows leave-raters-out splits: we fit modules on 20 raters and calibrate the remaining 5 from $n \in \{1,3,5,10,20\}$ images. Distribution metrics use QUBIQ, specifically Q-Dice and continuous Dice. On the brain-growth task we compare against the reported 0.5372 for the variational model and 0.5083 for Bayes dropout from Variational Inference for Quantifying Inter-observer Variability. We add the QUBIQ Q-Dice protocol and per-rater Dice where rater identity is preserved.

Three checks decide whether the modules are real rather than artifacts.

The first is planted recovery. We simulate raters on QUBIQ images by applying known mixtures of descriptor-conditioned dilations, for example expansion only where the gradient is low. NMF must recover the planted modules (matched cosine similarity above 0.9) where PCA does not.

The second is conservation, the analogue of the treadmill-versus-overground result. Modules fit on one half of the images, or on one QUBIQ organ task, should match those fit on the other half or another task. Each rater's recruitment vector should also correlate across halves more strongly than the vectors of different raters do.

The third separates image-driven from rater-driven structure. We permute rater labels within each image and refit. If the modules survive unchanged while rater recruitment stability collapses to chance, the modules reflect image ambiguity rather than style. That outcome would itself be informative, and we report which side of the split each module falls on.