Automatic segmentation of thin structures such as neurons, vessels and cracks tends to fail in one particular way. The structure is found, but in pieces. A single neuron in a 3D microscopy volume comes out as a dozen disconnected fragments, and someone has to stitch them back together by hand. Today the best automatic stitchers are classifiers trained on very large sets of human-proofread fragment pairs. This note proposes learning to stitch with no proofread pairs at all. We borrow an old trick from unsupervised translation. There, a model learns which letters in one language correspond to which letters in another by treating the correspondence as a hidden variable. It alternates between guessing the correspondence softly and re-estimating the letter-substitution statistics from those guesses (Expectation-Maximization, EM). Here the "letters" are fragment endpoints. The hidden variable is which other endpoint, if any, each endpoint truly continues into. The statistics being learned describe what a real break looks like: its gap length, bend angle, faint intensity along the bridge, and matching color.
The opportunity is concrete. On the FISBe benchmark of fly neurons in light microscopy, the best tested method reaches an aggregate score of only 0.35, and its dominant errors are false splits and false merges. Supervised stitching on electron microscopy (Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing) reaches 0.952 F1, but it needs 1.6 million proofread connections, which almost no imaging lab has. An unsupervised stitcher that runs after any segmenter, learns from the lab's own unlabeled volumes, and closes most of that gap would be directly useful. It would also be a clean test of whether alignment-with-a-null-option transfers from strings to space.
Core Claim
Reconnecting fragments has the same structure as unsupervised character alignment, and three ingredients from that literature carry over and do real work. The first is a null alignment: an endpoint is allowed to be a genuine termination rather than a break, just as a letter may align to nothing. The second is EM over soft link assignments: the model learns what a real continuation looks like from unlabeled data, starting from a weak geometric prior, in the way identity initialization seeds character alignment. The third is bidirectional agreement: the forward and backward alignments are combined so that a link is trusted when both endpoints choose each other, as in character alignment for Ethio-Eritrean Semitic languages, where combining both directions raised recall.
The claim is falsifiable in a specific way. If EM does not improve on its own iteration-zero geometric prior, then the alignment machinery adds nothing over a hand-tuned linking rule, and the idea fails.
Model
A base segmenter produces fragments. We extract skeleton endpoints. For each endpoint $a$ we gather candidate partners $N(a)$ within a radius, and connect $a$ to each candidate $b$ by a minimal-intensity-cost bridge path. Along each bridge we compute a feature vector $x_{ab}$ with five parts:
- gap length;
- tangent mismatch between the two fragment ends;
- mean and minimum image intensity along the bridge;
- distance in color-channel space between the two fragments;
- the local fragment density.
The color feature matters because FISBe images are multi-color, and fragments of one neuron share a color mix. Color plays the role that letter identity plays in transliteration.
Each endpoint has a latent choice $z_a \in \{\varnothing\} \cup N(a)$, where $\varnothing$ means the endpoint truly terminates. The generative story follows IBM-Model-1-with-null:
- the endpoint terminates with probability $\lambda$;
- otherwise it continues to one candidate chosen uniformly;
- the true partner's features are drawn from a "link" density $p_1$;
- all other candidates' features are drawn from a "non-link" density $p_0$.
The E-step posterior is then
$$ q_a(b) \propto \frac{1-\lambda}{|N(a)|}\,\frac{p_1(x_{ab})}{p_0(x_{ab})}, \qquad q_a(\varnothing) \propto \lambda . $$
Here $q_a$ is the soft assignment of endpoint $a$ over its candidates and the null option. The M-step re-fits three quantities using these fractional counts:
- $p_1$, from posterior-weighted features, as a small mixture model or a shallow monotone-constrained classifier;
- $p_0$, from the complementary weights;
- $\lambda$.
We initialize $p_1$ with a geometric prior that favors short, straight, bright gaps. This is the analogue of the identity-favoring initialization in Gasser et al. (2020).
Decoding keeps a link only if $a$ and $b$ each place more than half their mass on the other. This mutual-agreement rule also enforces that each endpoint has at most one partner without an intractable E-step over full matchings. One known failure from the string setting is expected to reappear. Tsai and Roth had to hand-prune the ten most frequent words so they would not corrupt the character parameters. The spatial counterpart is dense crossing regions and cell bodies, which generate many confident-looking spurious candidates. We handle this inside the model rather than by hand: local candidate density enters $p_0$ as a feature, and we report how much this matters.
Research Program
The work has three phases.
The first phase builds the pipeline on FISBe. The base segmenters are PatchPerPix, which is FISBe's strongest reported method, and a U-Net trained with soft-clDice, a supervised loss that rewards overlap of predicted and true centerlines. The stitcher runs after either segmenter.
The second phase is analysis. We check whether EM's learned $p_1$ recovers interpretable structure, such as a gap-length distribution peaked at signal-dropout scales and a strong reliance on color agreement. We check whether the null probability $\lambda$ converges near the true termination rate measured on held-out annotations. We also measure how sensitive the result is to initialization, since EM on discrete alignments is known to get stuck in local optima.
The third phase tests transfer. We run the identical method on FlyTracing's fragment pairs, using the provided pair labels only for scoring. This places an unsupervised stitcher on the same axis as the supervised one.
Baselines
The do-nothing reference is the base segmenter's raw fragments with no stitching. On FISBe the strongest such method is PatchPerPix, for which FISBe reports an aggregate score S = 0.35 and avF1 = 0.34 on the test set. Here avF1 is an instance-matching F1 computed with clDice, and S averages it with ground-truth coverage. We rerun PatchPerPix ourselves so that the stitched and unstitched numbers come from the same fragments. Beating this baseline means a clear gain in S that comes mainly from coverage and split reduction, with no rise in false merges.
The iteration-zero ablation is the exact same pipeline with EM disabled, linking by the initial geometric prior plus mutual agreement. This is the most important comparison because it isolates the learned component. To be fair, it receives the same features, the same candidate radius, and its agreement threshold tuned on the FISBe validation split with the same budget as ours. If EM does not beat it on S and avF1, we abandon the claim.
The hand-tuned linking rule applies thresholds on gap length, angle and bridge intensity, grid-searched on validation labels. This is what a practitioner would build first. Note that it consumes labels that our method does not.
The synthetic-break self-supervised classifier is the most dangerous competitor. We take long, confident fragments, cut them artificially, and train a classifier to rejoin the halves. It gets the same features, the same unlabeled volumes and the same compute.
The supervised ceiling is reported only on FlyTracing. Chen et al. report 0.952 F1 for their multimodal model and 0.891 for a shape-only model, both trained on 1.6M proofread pairs. Reaching the shape-only figure without any labels would count as a strong result. We do not expect to match 0.952.
Two further ablations remove the null option (every endpoint must link) and the bidirectional agreement (forward assignments only).
Why not just…?
Why not just use a stronger segmenter, such as a foundation model or a plain transformer? The evidence says scale does not remove the failure. SpeHeaTal reports that SAM, a general-purpose prompted segmenter, fails on thin sperm tails without added geometric help. SimpleClick, an interactive segmenter built on plain transformers with constant feature resolution, still struggles with extremely thin objects. In any case our stitcher runs after the segmenter, so a better segmenter is a complement rather than a rival. We test this by applying the stitcher to both the PatchPerPix and the clDice-U-Net outputs.
Why not just train with a topology-aware loss? Soft-clDice needs dense ground-truth skeletons. More importantly, it acts only within the network's receptive field during training. It cannot reconnect a neuron across a long signal dropout seen at test time, and FISBe's evaluation already scores with clDice-based matching while the best reported S stays at 0.35. The comparison between the clDice-U-Net with and without our stitcher measures exactly what is left over.
Why not just learn linking from synthetic cuts? This is a real threat, and the proposal is built around settling it. Synthetic cuts are free and give clean labels. Their weakness is that the cut locations do not follow the real break distribution: real splits concentrate at dim stretches, crossings and partial-volume regions. EM learns from the real breaks present in the data. Two experiments decide between them: a head-to-head comparison, and a hybrid that uses the synthetic-cut classifier as EM's initialization. If the synthetic classifier alone matches EM, the right conclusion is that self-supervision by corruption suffices, and we will report it that way.
Why not just tune shortest-path linking by hand? For sparse volumes it may be enough. The question is whether it holds up in dense volumes with more than 50 neurons, where geometry is ambiguous and color must be weighted against distance. We report results stratified by neuron density so that any regime where the heuristic suffices is visible.
Proposed Experiments
The primary experiment is on FISBe. We report S, avF1, coverage, and counts of false splits and false merges, stratified by the number of neurons per volume. Because the dataset has only 101 volumes, we compute confidence intervals with a bootstrap over volumes.
The secondary experiment is on FlyTracing. We compute pair-classification F1 from the EM posteriors with the agreement rule, and place it against the reported 0.891 and 0.952.
For a cheap 2D sanity check we use DRIVE and Massachusetts Roads, with segmenters trained on deliberately reduced data so that they fragment. No reported result found on Lacuna for clDice's numerical scores on DRIVE or Roads, so we generate these baselines ourselves.
Three checks decide whether the claim holds:
- EM over its own initialization. EM must improve S over the iteration-zero ablation. If it does not, the transfer of alignment ideas has failed.
- Calibrated null rate. The learned $\lambda$ must track the annotated termination rate. This shows the null option is modeling something real and not merely absorbing errors.
- Real breaks over synthetic ones. EM must beat or complement the synthetic-cut classifier specifically on dim and crossing breaks. This is the one place where learning from real breaks should matter.
References
- FISBe: A real-world benchmark dataset for instance segmentation of long-range thin filamentous structures
- Learning Multimodal Volumetric Features for Large-Scale Neuron Tracing
- Character Alignment in Morphologically Complex Translation Sets for Related Languages
- Learning Better Name Translation for Cross-Lingual Wikification
- clDice - A Novel Topology-Preserving Loss Function for Tubular Structure Segmentation
- SpeHeaTal: A Cluster-Enhanced Segmentation Method for Sperm Morphology Analysis
- SimpleClick: Interactive Image Segmentation with Simple Vision Transformers