Serial-section microscopy images a block of brain tissue one thin slice at a time. Each slice is imaged and then shaved away, producing a 3D stack that is fine in-plane and coarse along the cutting axis. Turning these stacks into wiring diagrams still depends on human proofreaders. They decide, fragment by fragment, whether two pieces of neurite belong to the same cell. These people often disagree. In whole-brain efforts the disagreement is serious enough that proofreaders are told to skip thin branches entirely, and FlyWire reports recovering only 44.6% of individual synapses as a result (FlyWire). Machine-learning work on annotator disagreement treats it as one quantity attached to a 2D image. Pionono, MRNet and DiFF all learn rater styles on fundus photos, ultrasound or histology crops. None of them asks the question that matters most in a volume: would the disagreement go away if the annotator could see more of the neuron?
We propose to measure and predict proofreader disagreement as a function of how much spatial context is visible. For each candidate merge decision in a 3D reconstruction, a model predicts how likely two independent experts are to disagree when shown a small, medium or large neighborhood. If predicted disagreement falls as context grows, the case is a context deficit and a machine with a wide view can settle it. If disagreement stays high at every scale, the ambiguity is real: the case should go to several humans or be kept as a probabilistic edge in the connectome rather than silently dropped. The multi-scale architecture is therefore not a route to slightly better segmentation. It is the measuring instrument that separates "the annotator couldn't see enough" from "the tissue does not determine the answer", which the inter-observer literature names as an unsolved problem.
Core Claim
Disagreement among proofreaders on serial-section volumes has scale structure. A hierarchical model trained on human votes collected under controlled context limits can predict that structure on unseen decisions. The prediction then supports two things a single consensus label cannot. The first is a split of disagreement into a part that more context resolves and a part it does not. The second is a proofreading triage policy that recovers more connectivity per hour of human effort than triage by model confidence alone.
For a candidate edge $e$ (a proposed join between two fragments) and a context radius $s$, let $p_s(e)$ be the fraction of raters who vote "same neuron" when shown context of radius $s$. Define the disagreement as $d_s(e) = 2p_s(e)(1-p_s(e))$, which is the probability that two randomly chosen raters disagree. The quantity we want is the resolvable gap
$$\Delta(e) = d_{s_{\min}}(e) - d_{s_{\max}}(e),$$
together with the residual $d_{s_{\max}}(e)$. A large $\Delta$ means a context deficit. A large residual means intrinsic ambiguity.
Relation to Prior Work
Three multi-rater methods supply the modeling vocabulary but stop at 2D and at a single scale. Pionono gives each annotator a Gaussian in a small latent space: the mean captures that annotator's bias and the variance captures their inconsistency. It injects samples once, before a U-Net decoder, on 2D histology. MRNet reconstructs each rater's mask and builds an uncertainty map on fundus and CT/MRI slices. DiFF weights raters by whichever fused mask best serves a downstream diagnosis. These papers motivate the project: they show that averaging raters throws away real alternative hypotheses. But all three treat disagreement as a fixed property of a pixel. We make it a function of the field of view, and we put it on topological decisions rather than boundary pixels. That second choice is forced by Analyzing Image Segmentation for Connectomics, which shows that voxel-overlap scores barely track whether the circuit is correct. Its Connectivity Correctness metric moved from 19% to 45% on the medulla between pipelines.
The connectomics proofreading literature supplies the decisions and the evaluation setting, and it constrains the design. Guided Proofreading found that searching for errors, not fixing them, is the bottleneck. It classifies split errors from fixed 75×75 2D patches against one curated reference. NEURD turns meshes into graphs of branch segments carrying features from spine counts up to branch angles, then flags merges with heuristics and graph networks. It handles false merges but not false splits. ConnectomeBench shows that frontier multimodal models working from 2D renderings reach only about 62% on merge identification. That supports the view that this decision needs volumetric, multi-scale evidence. None of these works treats disagreement among proofreaders as something to model.
The hierarchical-context literature supplies the architecture. PTree-Net and the pyramid graph network link nested coarse and fine patches with parent-child edges. DAS-MIL adds distillation between scales. Shrivastava et al. found that flat concatenation of multi-zoom features often fails to beat a single scale, a negative result that rules out the naive design. For serial EM specifically, the Residual Symmetric U-Net handles anisotropy by keeping full resolution along the cutting axis, using 2D convolutions at fine scales and 3D at coarse ones, and adding long-range affinities out to 27 pixels. TokenUnify pretrains a hierarchical state-space encoder on over 1 TB of EM. We reuse both: the anisotropy design for fine levels and TokenUnify-style pretraining for the encoder. Both of those papers treat the label as deterministic, which is precisely the assumption we drop.
Research Program
The first component is a controlled context-limited annotation study, which is also the dataset contribution. No multi-rater 3D serial-EM benchmark appears in the material we surveyed, so we build one. We start from an affinity-plus-watershed over-segmentation of SNEMI3D, CREMI and a MICrONS subvolume, and sample candidate edges between adjacent fragments. Sampling is stratified by fragment thickness, distance to known imaging defects such as missing or misaligned slices, and agreement with the existing reference. Each edge is shown to five or more trained proofreaders at three nested context radii. The smallest view is a few hundred nanometers around the contact, roughly what a 2D patch classifier sees. The middle view is a few microns in 3D. The largest is the full skeletonized arbors of both fragments together with the local EM. Each rater sees an edge at only one radius, so later views cannot contaminate earlier ones. As a secondary and weaker signal, we mine FlyWire's timestamped edit history for edits that one user made and a different user later reverted. These give naturally occurring disagreements at whatever context the proofreader chose. We use them only for external validation, because the context in those cases is uncontrolled.
The second component is a nested-context predictor. Level 1 sees only the smallest crop and uses an anisotropic 2D/3D encoder initialized from TokenUnify-style pretraining. Level 2 receives level 1's tokens plus a downsampled 3D neighborhood, through parent-child links as in PTree-Net. Level 3 adds a NEURD-style branch graph of both fragments. The design is cumulative: level $k$ conditions on everything up to $k$, and each level has its own output head predicting $p_s(e)$. Each head is trained with a binomial likelihood on the rater votes at its matching radius. Edges with only full-context votes supervise only level 3. This deep supervision per level answers the directional asymmetry raised in the multi-scale literature. A coarse level can never overwrite what a fine level predicts on its own, because the fine prediction is a separate, separately evaluated output. A per-rater latent offset in the style of Pionono is added at every level, so that annotator bias is not mistaken for scale-dependent ambiguity.
The third component is analysis and triage. From the predicted curve we estimate $\Delta(e)$ and the residual disagreement for every edge in a held-out volume. We test where irreducible ambiguity concentrates, hypothesizing near knife-defect slices and thin twigs. Edges are then routed to one of three outcomes: automatic acceptance by the level-3 decision, single review, or multi-rater review with the edge kept as probabilistic if raters still split. The policy is scored by connectivity recovered per simulated proofreading hour.
Baselines
The naive reference is a single-consensus pipeline. Majority vote over the full-context raters gives one label per edge. A classifier with the same encoder and the same largest context is trained on that label, and its predicted probability is read as a stand-in for disagreement. This tests whether our structure adds anything over "big model, clean labels". We will report its Brier score against held-out human vote fractions and its triage curve.
The strongest existing edge-decision method is the Guided Proofreading classifier. It reports 94% test accuracy for split-error classification on the L. Cylinder dataset (Guided Proofreading). We retrain it on our edges with its native 2D patch input, using the same training split and the same hyperparameter search budget as our model. NEURD is the merge-side reference. Its authors report 95–99% agreement with expert proofreaders and an increase in excitatory axonal synapse precision from 0.13 to 0.87 on MICrONS (NEURD). We run it on the MICrONS subvolume as published, as an off-the-shelf comparison for merge flags. ConnectomeBench's roughly 62% merge accuracy for o4-mini gives a floor for general models.
For disagreement prediction itself, the strongest competitor is a 3D port of Pionono: per-rater latents injected once before the decoder, trained on all votes pooled across radii. We found no reported result on Lacuna for any multi-rater method on 3D serial-section data, so this comparison is new and we run it ourselves under a matched compute and tuning budget. Three ablations isolate the key ingredient. The first trains our architecture with flat multi-scale concatenation instead of nested parent-child links. The second removes the per-level heads and keeps only the top one, which tests whether scale-indexed supervision matters. The third replaces human votes with ensemble variance computed at each level, which tests whether human supervision matters at all.
Beating the baselines means three things. First, lower held-out Brier score and higher rank correlation between predicted and observed $d_s$ at every radius than both the consensus model and 3D Pionono. Second, predicted $\Delta$ that correlates with observed $\Delta$ on held-out edges, which no single-scale baseline can produce. Third, a triage curve that dominates confidence-based triage in Connectivity Correctness, or in synapses attached, per review hour.
Why not just…?
Why not just train one large-context classifier on consensus labels and treat disagreement as label noise? If disagreement were mostly context deficit, this would be close to enough, and the measured $\Delta$ distribution will tell us. Pionono's comparison against STAPLE on Gleason 2019 (Cohen's κ 0.758 versus 0.750) and MRNet's gain over majority vote suggest that collapsing raters loses information, but those are 2D boundary tasks with small margins. The real test is the residual disagreement at full context. If it is near zero for almost all edges, the consensus baseline wins and we will say so. FlyWire's decision to drop orphan twigs suggests it will not be near zero for thin processes.
Why not just port Pionono or MRNet to 3D? That is the obvious move, and we run it as the main disagreement baseline. Its limitation is structural rather than a matter of tuning. A single injection point and labels pooled across views cannot express that the same edge is contested at one radius and settled at another, so it cannot output $\Delta$. It could still predict full-context disagreement as well as our model. If it does, the scale-indexed heads contributed nothing beyond the analysis, and the second ablation will reveal that.
Why not just use model ensemble uncertainty at several scales and skip expensive human labels? This is the most serious threat. Ensemble variance at each nested level could trace a resolvability curve without any rater data. The third ablation tests it directly. If ensemble-derived $\Delta$ matches human $\Delta$ as well as our supervised predictor does, the method claim collapses into the measurement study. That would still be useful: it would license cheap triage. We expect divergence where models and humans fail differently. Near defect slices, for example, a model may be confidently wrong while humans hesitate.
Why not just concatenate features from several resolutions? The negative result of Shrivastava et al. and the gains of the pyramid graph (70% versus 50–60% on glioma grading) argue against it in histology. Serial EM differs, so the first ablation checks this rather than assuming it.
Decisive Checks
The project succeeds if three things hold. Human disagreement measured on held-out edges falls with context for a substantial subset of edges and stays high for another identifiable subset. The nested predictor recovers both groups better than 3D Pionono and the ensemble ablation. Triage based on the split recovers more connectivity per proofreading hour than confidence-based triage on MICrONS and FlyWire-derived edits. Two results would falsify or sharply narrow the claim. The first is near-zero residual disagreement at full context, meaning disagreement is all context deficit and a big model suffices. The second is that ensemble variance reproduces human $\Delta$, meaning human supervision is unnecessary. A further sanity check is whether predicted irreducible ambiguity clusters at imaging defects and thin twigs. That would tie the learned quantity to a physical cause, not to rater idiosyncrasy already absorbed by the per-rater latents.
References
- FlyWire: online community for whole-brain connectomics
- Probabilistic Modeling of Inter- and Intra-observer Variability in Medical Image Segmentation
- Learning Calibrated Medical Image Segmentation via Multi-Rater Agreement Modeling
- Calibrate the Inter-Observer Segmentation Uncertainty via Diagnosis-First Principle
- Analyzing Image Segmentation for Connectomics
- Guided Proofreading of Automatic Segmentations for Connectomics
- NEURD offers automated proofreading and feature extraction for connectomics
- ConnectomeBench: Can LLMs proofread the connectome?
- Diagnose Like A Pathologist: Weakly-Supervised Pathologist-Tree Network for Slide-Level Immunohistochemical Scoring
- A Multi-scale Graph Network with Multi-head Attention for Histopathology Image Diagnosisn
- A Graph-Based Multi-Scale Approach With Knowledge Distillation for WSI Classification
- Deep Learning for Visual Recognition of Environmental Enteropathy and Celiac Disease
- Superhuman Accuracy on the SNEMI3D Connectomics Challenge
- TokenUnify: Scalable Autoregressive Pretraining for Large Scale EM Image Segmentation