When several pathologists outline the same cell in a stained tissue image, they mostly agree on the nucleus, the dark, high-contrast core. They disagree about where the faint cytoplasm ends, and most of all about where one cell stops and its neighbour begins. We propose to describe each cell's outline as a set of distances measured outward from its own nucleus, along rays at fixed angles. In that coordinate system, expert disagreement stops looking like diffuse pixel noise. It becomes a small number of interpretable quantities: how far a rater tends to push the outline, how uncertain the image itself is along each ray, and where a rater places a shared contact interface. The payoff is twofold. First, segmentation of overlapping cells gets calibrated uncertainty where it matters. Second, a new clinician's drawing style can be estimated from a handful of their annotated cells, without retraining.
This sits at a gap the literature leaves open. Methods that anchor faint structures on salient ones assume one correct answer. Multi-rater methods model disagreement only for single, non-overlapping regions. Nobody models how disagreement is distributed across the touching and overlapping interfaces of many instances.
Core Claim
Our claim is that inter-observer variability in whole-cell segmentation is anchor-structured. It should be low near the nucleus, grow with radial distance, and concentrate on rays that end at a contact or overlap interface with another cell. If that holds, modelling disagreement in nucleus-centred polar coordinates should beat modelling it in pixel space on three counts: calibration at interfaces, faithfulness of cell shape, and few-shot adaptation to unseen raters.
Concretely, for cell $i$ with nucleus centre $c_i$, rater $a$'s boundary distance along angle $\theta$ is modelled as
$$\log r^{(a)}_i(\theta) = \mu_i(\theta) + b_a^{\top}\phi_i(\theta) + \sigma_i(\theta)\,\epsilon_i(\theta).$$
The terms are as follows. $\mu_i(\theta)$ is the image-predicted typical boundary distance. $\phi_i(\theta)$ is a short descriptor of the ray, recording whether it ends at free background, at a contact with a neighbour, or at a translucent overlap, together with its normalized length. $b_a$ is a low-dimensional style vector for rater $a$. $\sigma_i(\theta)$ is the image's own ambiguity along that ray. $\epsilon_i(\theta)$ is standard normal noise.
For touching, non-overlapping cells, the opposing rays of neighbours $i$ and $j$ share one noise draw. This makes a rater's shift of a shared wall move both cells together rather than opening gaps. For translucent overlaps, the rays may cross and are sampled independently.
The equation separates image ambiguity from rater preference, in the spirit of ProSeg. ProSeg does this with two latent variables over whole semantic masks of lung nodules and tumours. We move the separation into per-instance anchor coordinates and couple it across neighbours. Because the model is linear in $b_a$ in log space, a new rater's style can be fitted in closed form from a few of their annotated cells.
Research Program
The model builds on nucleus anchoring as done in Cyto R-CNN. That method detects nuclei and then segments the whole cell inside a box enlarged by a fixed factor of 2.0. Its authors note that this fixed factor fails for elongated and pleomorphic cells.
We keep the nucleus branch but replace the box-plus-mask head with a head that predicts three things per ray: $\mu$, $\sigma$ and the ray-type probabilities. The cell's extent is therefore set by the predicted rays rather than a fixed box, and each instance owns its outline. Overlapping cytoplasms are representable directly. This sidesteps the suppression step that IRNet showed discards genuinely stacked cells. A lightweight attention layer across neighbouring nuclei, borrowing IRNet's idea of passing features from clear cells to faint neighbours, predicts the ray types and the coupling pattern.
The method assumes cytoplasm is star-shaped as seen from the nucleus centre. That assumption is a real risk for cells with non-convex extensions. The first step of the program therefore measures, on existing annotations, what fraction of cells are star-shaped from their nucleus centroid, and how much IoU is lost by the best polar approximation.
The main enabling contribution is data. Neither CytoNuke nor the Cervical Pap Smear dataset (CPS) has multiple raters, and we found no instance-level multi-rater whole-cell benchmark in the literature. We will have several pathologists independently re-annotate held-out subsets of both. CytoNuke is H&E head-and-neck carcinoma, with touching cells and faint membranes. CPS is cervical cytology, with translucent overlap. Raters work from shared nucleus seeds, so that cell identity is fixed and disagreement is purely about outlines.
Variability Matters found a Fleiss' kappa of only 0.43 among 120 pathologists on cell detection, and showed that low-conformity raters can hurt training. We will record per-rater conformity so that curation can be tested as a competing explanation.
Training uses the per-rater annotations, maximizing the likelihood of each rater's ray lengths under the equation above, with rater style vectors learned jointly for training raters. At test time, an unseen rater's $b_a$ is fitted from $k$ of their cells.
Baselines
The naive reference is a deterministic nucleus-anchored model trained on majority-vote masks, namely Cyto R-CNN as published. The file reports it at 58.65% whole-cell AP50 on CytoNuke, versus 45.33% for StarDist and 31.85% for Cellpose, with a morphology Kolmogorov–Smirnov statistic of D = 0.15. Any proposal that loses consensus accuracy against this is not worth its complexity. We therefore require non-inferior AP50 on the consensus test set and a lower morphology D statistic. On CPS the strongest reported overlapping-cell method is IRNet, at cytoplasm AJI 0.7185 and F1 0.7497, against Mask R-CNN at 0.6845 and 0.6664. Our consensus performance must match IRNet within noise.
The strongest competitor on the actual question is a pixel-space rater-latent model grafted onto the same anchored detector. This means Pionono's per-rater Gaussian latent concatenated into the Cyto R-CNN mask head, and separately ProSeg's two-latent formulation. Both are the leading multi-rater methods, and this graft is the obvious combination our idea must beat. Pionono reports κ = 0.758 on Gleason 2019, but no reported result found on Lacuna for any multi-rater method at the instance level on whole cells. All grafts will share the backbone, nucleus branch, data and training schedule, with an equal hyperparameter search budget. Unseen raters are handled by re-estimating the latent from the same $k$ cells by gradient steps.
A deep ensemble of deterministic anchored models, one per rater or rater split, is included because the QUBIQ challenge found such ensembles outperform single-model probabilistic networks. We will report its compute alongside its accuracy.
Two ablations isolate our ingredients. The first keeps the polar head but removes neighbour coupling and ray types. The second keeps everything but predicts $b_a$ as a pixel-wise offset map instead of a per-ray linear term. Beating the pixel-latent grafts on interface calibration and few-shot personalization, while matching consensus accuracy, is the success criterion.
Why not just…?
Why not just ensemble deterministic models, as QUBIQ recommends? This is a real threat on calibration alone, and the QUBIQ result may well repeat here. Our distinct claims are elsewhere. An ensemble of rater-specific models cannot represent a rater absent from training, and its members' interfaces are not coupled, so sampled neighbours can overlap or gap inconsistently. The experiments report interface calibration and few-shot personalization separately. If the ensemble matches us on both, we will report that the contribution reduces to the dataset and the polar representation.
Why not just curate high-conformity raters and train one deterministic model? Variability Matters shows curation improves the mean model, and it may improve our consensus AP as well; we will test it. But curation discards exactly the systematic styles that clinical deployment has to accommodate, and it yields no per-interface uncertainty. We will train our model and the naive baseline on curated and full rater sets to check whether our style term absorbs what curation throws away.
Why not just prompt a segmentation foundation model with nucleus points? Point-prompted foundation models are a form of anchoring, so this is a natural competitor. The PEFT no-go study reports that frozen Cellpose-SAM with parameter-efficient adapters fails at fine-grained dense instance delineation on LIVECell. We include nucleus-prompted Cellpose-SAM with adapters under the same budget, to see whether that negative result transfers to bright-field whole cells.
Why not just widen Cyto R-CNN's box adaptively? A learned box scale fixes clipping but not overlap, interface assignment or rater style. The first ablation, a polar head without the rater model, measures how much of the gain comes from adaptive extent alone.
Proposed Experiments
The decisive first check comes before any modelling. On the new multi-rater subsets, we will regress per-ray rater variance on radial distance and ray type. If disagreement is not markedly higher on contact and overlap rays than on free-boundary rays, and not increasing with distance, the core claim fails and we stop.
If it holds, the main comparison reports four things against every baseline:
- Consensus AP50 and AJI.
- Generalized energy distance between predicted and rater outline sets, computed per instance.
- Coverage of predicted ray-length intervals on held-out raters, separated by ray type.
- Few-shot personalization: IoU with an unseen rater's masks after fitting from $k$ cells, with $k$ ranging from 1 to 20, using leave-one-rater-out splits.
We expect the style vector to be identifiable from few cells, because it enters linearly with only a few dimensions. If so, personalization should saturate at small $k$ for our model but not for the pixel-latent grafts.
Two stress tests complete the study. First, we perturb nucleus positions to measure how anchor errors propagate into outlines, testing the one-way error propagation the file attributes to anchoring pipelines. Second, we evaluate on the densest CPS clumps to test whether the neighbour-coupled model degrades more gracefully than IRNet.
References
- Probabilistic Modeling of Multi-rater Medical Image Segmentation for Diversity and Personalization
- Cyto R-CNN and CytoNuke Dataset: Towards reliable whole-cell segmentation in bright-field histological images
- IRNet: Instance Relation Network for Overlapping Cervical Cell Segmentation
- Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- QUBIQ: Uncertainty Quantification for Biomedical Image Segmentation Challenge
- Is PEFT Enough for Cell Segmentation? An Empirical No-Go Result on Frozen Foundation Models