When several pathologists outline the same tumor, they disagree in consistent ways. One draws tight boundaries, another grades more aggressively, a third skips small glands. These habits carry clinical meaning. Methods that model them, however, need two things that are often missing: images paired with their masks, and a record of which rater drew each mask. Archives tend to hold a pile of slides and a separate pile of past delineations from many unnamed experts. This note proposes to learn, from such unpaired and unattributed piles, a generator that turns an image into a mask, where a small latent "observer code" chooses whose way of seeing to imitate. The question is whether the systematic habits of individual raters can be recovered without ever being told who the raters are.
The bet is that in a mask domain, "style" cannot mean color or texture, because masks have neither. Any per-sample latent factor that a translator learns about masks must therefore encode structural decisions: where a boundary falls, which grade a region gets, what counts as a lesion. If that latent space has a mixture prior, with one cluster per putative rater and a spread within each cluster, then the clusters should line up with real raters' systematic bias and the spread with their session-to-session inconsistency. This mirrors the supervised model of Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation (Pionono), but it is learned with no rater IDs and no pairs.
Core Claim
Unpaired image-to-image translation learns a mapping between two image collections that are not matched one-to-one. A translator trained this way into a pool of masks from many raters will fall into one of two behaviors. If it is deterministic, it collapses the raters into one averaged pseudo-rater. If it is given a latent code with a structured mixture prior, it separates them into identifiable rater-specific modes. The separation should be measurable directly, because on datasets where rater IDs exist we can hide them during training and reveal them at evaluation. The new element is the combination of three things: unpaired training, absent rater identities, and a latent space that is forced to control structural disagreement rather than appearance. No paper in the file does all three.
Relation to Prior Work
Two papers motivate the gap. The Consensus Trap surveys 346 annotation papers and argues that majority voting erases minority perspectives and produces pseudo-labels that match no real annotator. Pionono shows the constructive side in histopathology. It embeds each known rater as a Gaussian in an 8-dimensional latent space, conditions a ResNet34 U-Net on samples from it, and reports unweighted Cohen's κ of 0.758 on Gleason 2019 against 0.750 for the prior best. Pionono needs every training mask tied to an image and to a rater index. We drop both requirements, and we use its mean-versus-variance reading of rater latents as our evaluation target.
Three papers supply the machinery, and each falls short in a specific way.
- Contrastive Learning for Unpaired Image-to-Image Translation (CUT) keeps image content intact without a backward generator. It does this by matching each output patch to the input patch at the same location, using other patches of the same image as negatives. It reaches 24.7 mAP on Cityscapes label-to-photo versus 20.4 for CycleGAN. It is deterministic, so it cannot represent several valid masks for one image.
- Multimodal Unsupervised Image-to-Image Translation (MUNIT) splits images into spatial content and a pooled style code, applies the style through adaptive instance normalization, and samples style from an isotropic Gaussian. That prior has no notion of discrete raters, and its injection point is designed for texture.
- Video Summarization by Learning From Unpaired Data is the closest bridge. It treats summarization as unpaired translation from raw videos to a separate pool of human summaries. It assumes that pool is homogeneous and outputs a single deterministic summary. Our proposal is the heterogeneous-target version of that idea.
Two papers constrain the design. CUT's argument that cycle consistency is ill-posed when the two domains carry unequal information applies with extra force here. Image-to-mask is one-to-many across raters, so we avoid cycle reconstruction of the image, unlike Cycle-SUM. The evaluation-crisis results also shape the setting. Rethinking the Evaluation of Video Summaries reports random summaries scoring F1 0.57 on TVSum. Performance over Random shows rankings swing with the data split. Together these explain why we do not make video summarization the primary testbed, even though it is the most obvious home for subjectivity. Its metrics cannot yet tell rater-faithful generation apart from chance. Histopathology offers rater IDs consistent across images and an established κ protocol.
Research Program
Data setting. We build the unpaired, unattributed setting from Gleason 2019, which has six pathologists, and the breast cancer segmentation set, which has 25 raters. We split the training images into two disjoint halves. From one half we keep only the images. From the other half we keep only the masks, strip their rater IDs, and shuffle them. This is the same 50/50 construction UnpairedVSN used on its 164-video pool. Test images keep all their attributed masks.
Model. The model has three parts:
- A generator $G(x, z)$ maps an image $x$ and an observer code $z$ to a mask.
- A mask encoder $E(y)$ maps any mask $y$ to a code.
- A mask discriminator judges realism.
Content is tied to the image by CUT's patchwise contrastive loss between $x$ and $G(x,z)$. The code enters the generator at the decision layers through spatially varying modulation of the features that feed the class logits. We do not use global adaptive normalization, because rater disagreement is about boundaries and grades.
The key ingredient. The key ingredient is the prior on $z$:
$$ p(z) = \sum_{k=1}^{K} \pi_k\, \mathcal{N}(z;\, \mu_k, \Sigma_k), $$
where $K$ is the number of putative raters, $\pi_k$ is how much of the mask pool component $k$ accounts for, $\mu_k$ is that component's systematic bias, and $\Sigma_k$ is its internal inconsistency. All of these are learned.
Training losses. Training combines four terms:
- the adversarial loss on $G(x,z)$ with $z \sim p(z)$;
- MUNIT-style code reconstruction, $\lVert E(G(x,z)) - z\rVert$, which stops the generator from ignoring $z$;
- a mixture-likelihood term pulling $E(y)$ of real masks toward $p(z)$;
- a code-swap term.
The code-swap term takes the code of a real mask $y'$ that was drawn for some other image and applies it to a new image $x$. The result must still look realistic and must re-encode to the same code. This pushes $z$ to carry what stays constant for a rater across images, and pushes everything image-specific into the content pathway. It is the main defense against the obvious confound, where masks of very cancerous tissue get mistaken for masks from an aggressive grader.
Reading the codes out. At test time a rater is represented by encoding a handful of that rater's attributed masks, by default five, and averaging their codes. We call this few-shot rater calibration. It is used only for evaluation and for deployment-time personalization, never in training.
Analysis. Before the real data, we run a controlled identifiability study. Starting from consensus masks, we simulate raters with known biases of adjustable strength: boundary dilation or erosion, shifts in grade, and dropping small components. We then measure how rater recovery degrades as the biases weaken and as $K$ grows. This shows when the claim can hold at all.
Baselines
Naive reference: pooled CUT. The do-nothing reference is CUT trained on the pooled, unattributed mask half and asked to produce one mask per image. It is the direct image-domain analogue of UnpairedVSN and represents current unpaired practice. We score it against every rater's masks. A pooled model can only match the average rater, so its per-rater κ marks the floor that our model's rater-conditioned outputs must clear. We report it alongside the consensus-oracle score, meaning the same pooled model evaluated against STAPLE consensus. That pairing shows how much the per-rater view moves the numbers.
Strongest unpaired competitor: MUNIT. MUNIT is trained on the same halves with the same ResNet34-based encoder and decoder capacity, the same number of iterations, and the same hyperparameter search budget of about twenty configurations per method, selected on a held-out unpaired validation split by realism plus code reconstruction. MUNIT uses its own isotropic prior. For a fair readout we fit a $K$-component mixture to its style codes after training and apply the same few-shot calibration. Beating it means higher normalized mutual information between codes and true rater IDs, and higher calibrated per-rater κ, across three seeds with non-overlapping intervals.
Supervised upper bound: Pionono. Pionono sees pairs and rater IDs, so it is a reference ceiling rather than a competitor. It reports κ 0.758 on Gleason 2019 and 0.711 on the breast cancer set. We re-run it on our exact splits so that any gap reflects supervision and not data differences. Success means closing a substantial share of the gap between pooled CUT and Pionono, decided in advance as at least half, without using pairs or IDs.
Ablations of the key ingredient. Three ablations isolate what matters:
- replacing the mixture prior with an isotropic Gaussian while keeping everything else, including decision-layer injection;
- moving code injection from the decision layers to global normalization;
- removing the code-swap term.
If the isotropic-prior ablation recovers raters as well as the full model, the mixture prior is not the active ingredient, and the claim narrows to "mask-domain codes are structural by necessity."
Why not just…?
Why not just aggregate with majority vote or STAPLE and train one model? That answers a different question, and Pionono's comparison against consensus-trained models already shows that collapsing raters loses accuracy on these datasets. More basically, a consensus model cannot emulate any particular pathologist, which is the deliverable here. The pooled-CUT baseline is this alternative in unpaired form. Its per-rater κ against its consensus κ directly measures what aggregation discards.
Why not just cluster the unattributed masks first, then train one unpaired translator per cluster? This is the real threat. If rater habits are visible from masks alone, plain clustering could recover raters cheaply. The weakness is that mask statistics mix rater habit with image content. A mask covering lots of high-grade tissue looks like an aggressive grader, and clustering cannot tell the two apart without the image. Our code-swap term exists to break that confound. We run this baseline in full: masks are encoded with a mask autoencoder of matched capacity and clustered with k-means, the clusters are matched to raters with the same few-shot calibration, and one CUT is trained per cluster with an equal total compute budget. The synthetic study settles the matter. We create biases that are confounded with content by construction, for example by giving the aggressive simulated rater the images with more tumor. If clustering matches our model even there, we will report that the translator adds nothing for identification and reposition the contribution as generation quality alone.
Why not just use MUNIT as-is and read raters off its style space? MUNIT pools style globally and assumes isotropic noise. Nothing in its objective favors a few discrete, image-invariant modes, so it may spread rater identity across the space or entangle it with lesion size. Whether this matters is empirical, which is why MUNIT is the strongest competitor and why the isotropic-prior ablation exists. A tie would be an honest negative result about the prior. It would not be a reason to hide the comparison.
Why not just do this in video summarization, where subjectivity is most visible? The current metrics on SumMe and TVSum cannot separate methods from random selection, per Otani et al. and the Performance-over-Random analysis. TVSum's crowdsourced annotators are also not tracked consistently across videos, so rater recovery cannot be scored. Histopathology gives us the ground truth that rater-discovery claims require. A video extension makes sense only after the method is validated, and it would use Otani et al.'s rank-correlation metrics rather than F1.
Proposed Experiments
Datasets. Gleason 2019 is the primary benchmark. The breast cancer set tests many raters with fewer masks each. Arvaniti TMA serves as external validation; the file reports no Lacuna result for it, so we report our own numbers only.
Primary metrics. Normalized mutual information and adjusted Rand index between learned code clusters and hidden rater IDs, and per-rater unweighted Cohen's κ under Pionono's protocol after five-shot calibration.
Secondary metrics. Consensus κ, so that rater-faithfulness is not bought with worse average accuracy. A test of the intra-observer reading: whether the learned per-component spread ranks raters in the same order as their measured self-inconsistency, wherever repeated annotations allow it.
Decisive checks. The first decisive check is the content-confounded synthetic study against the cluster-then-translate baseline. The second is the comparison of calibrated per-rater κ against pooled CUT and against our re-run of Pionono on identical splits. The third is a sweep of $K$ above and below the true rater count, which tests whether the method degrades gracefully or invents raters. The deployment claim is that a new expert's habits can be imitated from five of their masks without retraining. We test it by holding one rater out of training entirely and calibrating on that rater at test time.
References
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation
- Contrastive Learning for Unpaired Image-to-Image Translation
- Multimodal Unsupervised Image-to-Image Translation
- Video Summarization by Learning From Unpaired Data
- Cycle-SUM: Cycle-Consistent Adversarial LSTM Networks for Unsupervised Video Summarization
- Rethinking the Evaluation of Video Summaries
- Performance over Random: A Robust Evaluation Protocol for Video Summarization Methods