Merge-Safe Sparsity for Connectomics Segmentation Networks

Mapping brain wiring from electron microscopy means running 3D neural networks over trillions of voxels. Humans then fix what the networks get wrong. The expensive part is the human fixing, called proofreading. One fruit-fly connectome took an estimated 33 human-years of it (ConnectomeBench). The worst errors are merges, where two distinct neurons are fused into one. Merges are harder to find and fix than splits, where one neuron is broken into pieces. Meanwhile, modern GPUs can nearly double throughput on networks whose weights follow "hardware-aligned" sparsity patterns. The main example is 2:4 sparsity, where every group of four weights keeps exactly two nonzeros, a pattern the Tensor Cores of recent NVIDIA chips execute natively. Such pruning is nearly lossless on ImageNet, but ImageNet accuracy does not tell you whether the pruned model separates touching membranes.

This note proposes to measure and then control that risk. We will first test whether hardware-aligned sparsity damages connectomics networks asymmetrically: does it raise merge errors much more than split errors, because thin, faint membrane boundaries are exactly what a capacity-starved network loses first? We will then build a pruning and allocation procedure that scores weights by their effect on boundary affinities between different objects, not by overall loss. That procedure picks, layer by layer, which parts of the network go 2:4, which go block-sparse, and which stay dense. We rank candidate configurations with a cheap proxy instead of full training. The deliverable is a segmentation network that runs faster on real hardware and adds no merge errors, and therefore no extra proofreading hours.

Core Claim

For reconstruction networks, the right compression target is merge error per unit of GPU time, not average loss or Rand score. We make two predictions. First, standard hardware-aligned pruning, applied uniformly, will leave the headline Rand error nearly unchanged while shifting errors toward merges. Second, a saliency score restricted to voxels on boundaries between different ground-truth objects will locate the weights and layers that must stay dense. This yields a mixed sparsity configuration that is as fast as uniform 2:4 and as merge-safe as the dense model.

Architecture search enters only as a constraint on cost. We search the small space of per-layer sparsity patterns and compensating widths ("prune to 2:4, then widen by some factor"). We do not search open-ended topologies, and we evaluate the ranking honestly against short-training baselines.

Relation to Prior Work

Two hardware-aligned pruning methods supply our machinery. Learning Best Combination for Efficient N:M Sparsity (LBC) learns which of the possible two-of-four patterns to keep in each block, and reports 77.2% ImageNet top-1 for 2:4 ResNet-50 against 77.1% dense. Spartan learns 16×16 block masks through a smoothed top-k selection (a differentiable relaxation that hardens into discrete blocks during training), and reaches 76.5% on ResNet-50 at 95% sparsity. Both optimize task loss on 2D classification. Neither has been tested on 3D affinity networks, meaning networks that predict whether neighboring voxels belong to the same object. Neither distinguishes an error that fuses two neurons from one that breaks a neuron. We reuse their optimizers and change what they score.

On the connectomics side, Superhuman Accuracy on the SNEMI3D Connectomics Challenge gives the dense reference model. That paper's Residual Symmetric U-Net has 1.5M parameters and a 0.02576 Rand error; the authors themselves warn against extrapolating the benchmark to large-scale reconstruction. An Error Detection and Correction Framework for Connectomics supplies a second, heavier network worth accelerating: its error detector and corrector cut 944 errors to 305 on mouse V1 in under 100 minutes per 2048×2048×256 volume on one GPU. PATHFINDER and NEURD motivate the merge focus. PATHFINDER's large gains came mainly from cutting merges tenfold, and its verification stage is expensive. NEURD's automated edits target false merges. ConnectomeBench shows merges are the error type even multimodal language models cannot find, at about 62% accuracy for the best model. Together these argue that a compression method that trades splits for merges quietly moves cost back onto humans.

For cheap search, Zero-Cost Proxies for Lightweight NAS repurposes pruning saliencies (snip, synflow, and others) to rank architectures from one minibatch. It reports a 0.82 Spearman correlation on NAS-Bench-201, against 0.61 for reduced training. It also reports that such proxies degrade across heterogeneous topologies, which is one reason we keep our search space narrow. Scalable NAS for 3D Medical Image Segmentation (SCNAS) shows that 3D search is feasible but costs roughly one day on four V100s. That cost sets the ceiling our search overhead must stay well below. None of these papers places sparsity masks inside the proxy calculation, and none evaluates on topology-sensitive error metrics.

Research Program

Stage 1: diagnosis. We take the dense Residual Symmetric U-Net trained on SNEMI3D and AC3, and the Zung et al. error detector trained on mouse V1. We prune each with uniform LBC 2:4, uniform Spartan 16×16 blocks at matched parameter budgets, and unstructured magnitude pruning. We then decompose the resulting errors into split and merge components, using the split and merge terms of variation of information and of Rand error after the same agglomeration. We also measure the precision and recall of the error detector. We locate where new merges occur by stratifying voxels by membrane contrast. The hypothesis is that new merges concentrate at faint or defective membranes, the weak-signal regime described by Identifying Weak Signals in Inhomogeneous Neuronal Images.

Stage 2: merge-weighted saliency. Let $B$ be the set of voxel pairs that straddle a boundary between different ground-truth objects, and let $\ell_B(w)$ be the affinity loss restricted to $B$. We score each weight $w_i$ by $$s_i = \left| w_i \, \frac{\partial \ell_B}{\partial w_i} \right| + \lambda \left| w_i \, \frac{\partial \ell}{\partial w_i} \right|,$$ where $\ell$ is the full loss and $\lambda$ trades general accuracy against boundary protection. This score replaces the magnitude or task-loss scores inside LBC's pattern selection and inside Spartan's cost vector. Summed per layer, the same score measures how much boundary-relevant signal each layer would lose under a given pattern.

Stage 3: constrained configuration search. The search space assigns each layer one of four choices: dense, 2:4, 16×16 block, or 2:4 with channels widened by 1.25× or 1.5×. The widening recovers capacity while keeping Tensor Core eligibility. We rank configurations three ways: by the Stage 2 score computed after projecting random-initialized weights onto each mask; by the same score after a few hundred training steps; and by short full training. To validate the rankings, we fully train about 100 configurations. The ranking target is merge error at a fixed measured latency. Whichever cheap ranker correlates best is the one we keep, even if it is not the zero-cost version.

Baselines

Do-nothing reference. The first baseline is the dense Residual Symmetric U-Net as published, with its reported SNEMI3D Rand error of 0.02576 (Superhuman Accuracy on SNEMI3D). On mouse V1, the reference is the dense Zung et al. pipeline and its 305 residual errors (Error Detection and Correction). We retrain both ourselves under our data splits so that numbers are comparable, and we report the published figures alongside. Any sparse model must match the dense model's merge error within seed variance to count as merge-safe.

Strongest existing method. The strongest competitor is uniform post-hoc 2:4 with LBC (Learning Best Combination). LBC is the best reported 2:4 method in the file (77.2% ResNet-50 top-1), it requires no search, and it already yields hardware speedups. We also run Spartan at matched density (Spartan). No reported result was found on Lacuna for either method on SNEMI3D, AC3, or any 3D EM benchmark, so we establish these numbers ourselves. For fairness, every method gets the same training data, the same total GPU-hours including any search cost, and the same hyperparameter sweep budget. Latency is measured on the same GPU with the same kernels. We beat LBC only if we reach equal or lower merge error at equal or lower measured latency, or lower latency at equal merge error, with the gap exceeding seed variance.

Accuracy ceiling without speed. Unstructured magnitude pruning at the same parameter count shows how much of any loss comes from the pattern constraint itself rather than from reduced capacity.

Ablations. The central ablation runs the identical Stage 2 and Stage 3 pipeline with the standard full-loss saliency in place of the boundary-restricted one. This isolates whether merge-weighted scoring matters. A second ablation fixes the per-layer allocation to uniform 2:4 while keeping the new saliency, which separates the within-layer gain from the allocation gain.

Why not just…?

Why not just apply LBC 2:4 to the existing network and stop? This is the real threat. If Stage 1 shows uniform 2:4 leaves the merge component unchanged, most of this proposal is unnecessary, and the right outcome is a short paper reporting that the cheap fix is safe for 3D EM. We treat that as an acceptable result. The Stage 1 split/merge decomposition settles the question directly before any method-building begins.

Why not just use unstructured pruning with a sparse library? Unstructured sparsity rarely speeds up GPUs, because irregular memory access and divergent threads cancel the saved arithmetic (Balanced Sparsity for Efficient DNN Inference on GPU; Dynamic N:M Sparse Attention). We include it anyway as an accuracy ceiling and report its measured wall-clock time. If it is somehow fast on 3D convolutions, that result would change our recommendation.

Why not just train each candidate briefly instead of trusting a zero-cost proxy? Zero-cost proxies are unvetted in 3D and known to fail across heterogeneous topologies (Zero-Cost Proxies for Lightweight NAS). Short training is therefore a first-class ranker in Stage 3, not a straw man. Because our search space holds only a few hundred configurations, short training may well be affordable and win. The proposal's contribution is the merge-weighted objective, not the zero-cost machinery.

Why not just buy more GPUs or rely on downstream proofreaders like NEURD? Downstream tools catch merges only partially. NEURD speeds manual merge-splitting threefold rather than eliminating it (NEURD), and merges resist even language-model proofreading (ConnectomeBench). Compression that adds merges shifts cost onto the most expensive stage of the pipeline. Compression that does not add merges is pure savings. We will report GPU-hours and projected proofreading edits together so the trade-off is explicit.

Proposed Experiments and Decisive Checks

The benchmarks are SNEMI3D and AC3 for affinity networks, and the Allen mouse V1 volume (2.3 Gvoxels) for the error detector and corrector. If we get access to the IBEAM-mSEM volume, we will add a run-length metric (nERL) there, since the file reports 94.2% for PATHFINDER axons. The metrics are Rand and variation of information, each split into merge and split terms; merge errors per unit reconstructed path length; detector precision and recall; and measured voxels per second on an Ampere or Hopper GPU using native 2:4 kernels.

One hardware caveat applies. At 1.5M parameters, the U-Net may be limited by memory bandwidth rather than arithmetic. If that holds, 2:4 would bring little speedup regardless of accuracy. So the first decisive check is a latency measurement of dense versus 2:4 3D convolutions at production patch sizes. A second check is whether 2:4 pruning raises the merge term of variation of information by a larger relative amount than the split term. A third is whether boundary-restricted saliency beats full-loss saliency in the central ablation at matched latency.

If a cheap ranker's Spearman correlation with fully trained merge error falls below that of short training, we drop the cheap ranker and say so. Success means one of two outcomes at no added merge error: a measured throughput gain over the dense network, or a demonstrated loss of merge safety under uniform pruning that our allocation recovers.