When biologists count lab-grown mini-organs (organoids) in brightfield microscopy, two experts often disagree. The most common dispute is whether a dark blob is one large organoid or several smaller ones stacked on top of each other. The MultiOrg dataset documents this "clump problem," along with annotators changing their own minds when they re-label the same images later (MultiOrg). The same dataset also built the problem in by design. To make labeling tractable, the 3D focal stacks were flattened into 2D maximum intensity projections, a single image that keeps the brightest value along each depth column. Organoids at different depths merge into one silhouette in that image. We propose to measure how much of the human disagreement is caused by this flattening. To do so, we train a lightweight detector that sees a few neighbouring focal planes instead of the projection, and we test whether its residual uncertainty separates disagreements that depth information resolves from disagreements that reflect real biological ambiguity.
The setting matters because brightfield imaging is the low-harm alternative to fluorescence. Fluorescence illumination produces reactive oxygen species that disturb or kill living cells (Grah et al.; Heaster and Skala). Brightfield is what makes long-term, repeated organoid imaging possible. Collecting a few extra brightfield focal planes costs the sample little, but discarding them at analysis time throws away exactly the information experts would need to settle clump disputes. If most clump disagreement can be recovered from depth, the right response is to change the input to the detector, not to build more elaborate models of annotator noise.
Core Claim
We claim that inter-observer disagreement in projected brightfield organoid images splits into two parts. The first is a depth-recoverable part, which disappears once a model or a human sees the focal planes around the object. The second is an irreducible part, which persists even with depth context. A pseudo-3D detector can estimate this split per object. Here pseudo-3D means a 2D network whose three input colour channels hold three neighbouring focal planes, following the context-aware pseudocoloring of EfficientCellSeg. The detector is trained with one output head per annotator, so it can predict where those annotators will disagree. The specific, falsifiable prediction has three parts. First, moving from projection to multi-plane input should improve detection mainly inside regions where annotators disagreed, not uniformly across the image. Second, the remaining disagreement between the two heads should predict which clumps humans still dispute after they are allowed to scroll through the focal stack. Third, scrambling the order of the planes should remove the gain.
This differs from the closest work as follows. MultiOrg combines multi-rater labels with 2D projections and never restores depth. EfficientCellSeg restores local depth cheaply but works on fluorescence data with a single consensus label. Pionono (Probabilistic Modeling of Inter- and Intra-observer Variability) models each rater's style but treats disagreement as a property of the raters, not of the input representation. Our work asks whether part of what looks like "rater style" is actually information the input removed.
Research Program
The detector feeds a 2D, ImageNet-pretrained backbone with three focal planes encoded as red, green and blue. Following EfficientCellSeg, the centre plane is the plane of best focus for the field, and contrast-limited adaptive histogram equalization (CLAHE) is applied to the two outer planes.
Brightfield stacks raise one complication that fluorescence stacks do not. The planes are defocus views, not optical sections, and the step between planes varies across MultiOrg's 26 studies. We therefore choose the outer planes by physical distance from focus, not by index. This addresses the anisotropy concern that plain channel stacking ignores. Five-plane and seven-plane variants pack the extra planes through a learned 1×1 input projection. This lets us measure how few planes are needed, which is the quantity that matters for keeping acquisition time short.
On the label side, we follow the virtual expert panel of More is more. A shared trunk feeds two detection heads, one per MultiOrg annotator. We prefer this over Pionono's latent rater space because MultiOrg has only two raters, and an 8-dimensional per-rater Gaussian has little to learn from two people. Head disagreement on a given object, computed through matched-box overlap and score difference, is our estimate of irreducible ambiguity.
There is one dependency we cannot resolve from the literature file. The method needs the raw MultiOrg focal stacks, and the file does not say whether these were released. We will request them from the authors. If they are unavailable, we will collect a smaller organoid brightfield stack set with two annotators who use the same protocol.
To test the decomposition on data where the answer is known, we add a controlled testbed built from the Allen Institute label-free dataset of paired 3D brightfield and fluorescence stacks with curated 3D nuclear masks, as used by Spotlight. We project the brightfield stacks and identify nucleus pairs that overlap in the projection but are separate in 3D. These are true depth-recoverable "clumps." Pairs that touch in 3D serve as genuinely ambiguous cases. Simulated raters then label the projected images with different merge thresholds. On this testbed we can check directly whether the pseudo-3D detector's head disagreement collapses on the depth-recoverable pairs and persists on the touching ones.
Baselines
Naive reference. The do-nothing reference is the MultiOrg pipeline as published: a 2D detector trained on projected images with single-annotator targets. The strongest reported number is SSD at [email protected] of 68.09% (MultiOrg). We will reproduce it with our own backbone on the projections, so every comparison differs only in the input. All inputs get the same training schedule, the same hyperparameter search budget, the same test split, and evaluation against each annotator's t⁰ labels and the re-annotated t¹ labels.
Best single 2D image. Our strongest 2D competitor gives the same detector the single best-focus plane, and separately an extended-depth-of-field fusion image. These are what a practical lab would try first.
Full 3D detector. We also run a 3D convolutional detector trained from scratch under matched GPU-hours. EfficientCellSeg reports that 3D ensembles reach comparable segmentation quality with 176M parameters against 6.7M. It also reports 4.4× slower CPU inference, 1.47 s against 334 ms. So the 3D model is a real accuracy competitor, and the question is whether it buys anything on clumps specifically.
Ablations isolating depth. The key ablation keeps the channel count and parameter count fixed but removes depth information. In one version all three channels hold the centre plane. In another the plane order is shuffled per sample.
Label-side baselines. Here we compare a consensus target (union and intersection of the two annotators), soft labels that average the annotators, and the two-head panel.
What counts as winning. Beating the baselines means two things. First, a statistically reliable mAP gain over the matched projection detector that is concentrated in disagreement regions; we require the gain in disagreement regions to be at least twice the gain elsewhere. Second, head-disagreement AUROC for predicting persistent human disputes that clearly exceeds the AUROC of ensemble variance from the projection model.
Why not just…?
Why not just use the best-focus slice or an extended-depth-of-field image with an ordinary 2D detector? This is the real threat. If a single well-chosen image separates stacked organoids as well as three planes do, the pseudo-3D input is unnecessary. We do not have evidence in hand that rules this out. The single-image baselines and the replicated-centre ablation exist to settle it. If they match the pseudo-3D detector on clump regions, we will report that depth ordering adds nothing. The disagreement-decomposition analysis would then be rerun on the best single image, because the scientific question about projection-induced disagreement still stands.
Why not just train a full 3D network? A 3D network sees more depth, but it gives up ImageNet pretraining and costs far more compute. MultiOrg's 411 images are a small training set. EfficientCellSeg's comparison suggests 2.5D input matches 3D at a fraction of the cost. However, that comparison was on fluorescence segmentation, not brightfield detection, so we run the 3D model under a matched budget and do not assume the result transfers.
Why not just merge annotators into a consensus or soft label and use ensemble variance for uncertainty? Consensus training can improve accuracy. Ensemble variance tracks prediction error well in virtual staining, with Pearson r≈0.83 in Trustworthy in silico cell labeling. But ensemble variance measures model uncertainty, not where humans split. The More is more panel found consensus training badly miscalibrated, with ECE 0.183 against 0.050 for the panel. We test this directly: if ensemble variance from a consensus model predicts persistent human disputes as well as head disagreement does, the per-rater heads are not needed and we will drop them.
Proposed Experiments and Decisive Checks
Human re-annotation. The decisive experiment is a small human study. We sample clump regions where the t⁰ annotations of annotators A and B disagree; a few hundred regions is the target. Both annotators re-label these regions while scrolling through the focal stack. We measure the fraction of disagreements that disappear, which is the human estimate of depth-recoverable ambiguity. We then ask whether the pseudo-3D detector's head disagreement predicts which disputes survive, measured by AUROC, against projection-model ensemble variance and against the t⁰→t¹ intra-rater changes reported in MultiOrg.
Error analysis by region. Detection results are reported as [email protected] and F1 against each annotator and against t¹, split into agreement and disagreement regions. We also report split and merge error counts on clumps.
Planes versus acquisition cost. A plane-count sweep over 3, 5 and 7 planes, with physical-distance selection compared against index-based selection, tells experimenters how many brightfield planes to acquire.
Controlled testbed. On the Allen projection testbed, we report the same AUROC separately for depth-separable and touching nucleus pairs, where the correct answer is known.
Failure condition. The hypothesis fails if the gains are uniform across regions, if shuffled planes perform as well as ordered ones, or if scrolling humans resolve few disputes. Any of these results would mean clump disagreement in organoid data is mostly not a projection artifact, which is itself worth knowing before the field invests further in modeling rater noise.
References
- MultiOrg: A Multi-rater Organoid-detection Dataset
- Mathematical imaging methods for mitosis analysis in live-cell phase contrast microscopy
- Quantitative Spatial Analysis of Metabolic Heterogeneity Across in vivo and in vitro Tumor Models
- EfficientCellSeg: Efficient Volumetric Cell Segmentation Using Context Aware Pseudocoloring
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- More is more: leveraging multi-rater information for whole slide images grading via virtual expert panel
- Foreground-aware Virtual Staining for Accurate 3D Cell Morphological Profiling
- Trustworthy in silico cell labeling via ensemble-based image translation