Where Does the Single Pass Fall Short? Measuring and Selectively Closing the Amortization Gap in Sparse-Label Mask Expansion

A common way to cut annotation cost is to have annotators click once per object, or label a few slices of a 3D scan, and let a model expand those sparse marks into full masks. There are two ways to do the expansion. One is to solve a small optimization problem for each image, for example a transport problem that assigns every pixel to one of the clicked points. The other is to train a network that maps the image and its clicks to masks in a single forward pass. The network is fast but systematically worse on individual images. In generative modeling this shortfall has a name, the amortization gap: the loss from asking one shared network to do what per-example optimization would do better. It is often the dominant error (Inference Suboptimality in Variational Autoencoders).

This note proposes measuring that gap in sparse-label mask expansion, per image and per region. We would split the total error into "the network failed to optimize the objective" and "the objective itself is wrong." We then test whether the first part is concentrated in a small, cheaply predictable subset of images. If it is, a fast network can run a few refinement steps only where they will pay off. That would recover most of the quality of per-image optimization at close to single-pass latency. It would also tell an annotation team, image by image, whether to spend more compute or ask for more clicks.

Core Claim

The setting is label expansion at deployment. A new image or volume arrives with sparse human marks: points per instance, or a few annotated slices. The goal is dense masks. Annotation tooling needs exactly this, and per-instance optimization is well defined here because real supervision is present at test time.

Let $E(m; x, s)$ be an expansion objective over a soft mask $m$ for image $x$ with sparse marks $s$. It combines a loss on the marked pixels with a propagation prior: the geodesic objectness of Point-supervised Segmentation of Microscopy Images and Volumes via Objectness Regularization, or the point-to-pixel transport cost of Point2Mask. A trained expander $f$ outputs $f(x,s)$ in one pass. Writing $m^\ast$ for the best mask found by long per-instance optimization, and $\mathrm{Err}$ for one minus IoU against ground truth, we use the decomposition

$$ \mathrm{Err}(f(x,s)) = \underbrace{\mathrm{Err}(f(x,s)) - \mathrm{Err}(m^\ast)}_{\text{amortization gap}} + \underbrace{\mathrm{Err}(m^\ast)}_{\text{objective (approximation) gap}} , $$

which mirrors the decomposition of Inference Suboptimality in Variational Autoencoders, but with task error in place of nats.

We make three claims. First, the amortization gap in expansion is heavy-tailed: most images are near-optimal after one pass, and a minority carry most of the gap. These are typically adjacent same-class instances and regions far from any seed. Second, it widens on images unlike the training set, as that paper and Zhang et al. and Zenn et al. report for encoders. Third, the size of the gap on a given image can be predicted from signals available after the first refinement step. Together these imply that gated semi-amortized refinement should beat both always-refine and never-refine on the quality-versus-latency curve.

Relation to Prior Work

The method ingredients come from semi-amortized inference. Semi-Amortized Variational Autoencoders initialize per-example optimization with an encoder and backpropagate through K unrolled steps; with K=10 they beat 80 steps of pure per-example optimization. Buffered Stochastic Variational Inference reuses the whole refinement trajectory instead of only its last state. CryoSPIN applies encoder-initialized refinement to cryo-EM pose estimation. All three refine every example for a fixed number of steps, and all three work on whole-example latents rather than dense masks. We borrow the encoder-as-initializer idea and add two things: per-instance gating, and a gap decomposition measured in task error.

The motivation comes from the refinement-cost literature. Lung-segmentation cascades (Hwang & Park) and feedback CNNs (Learning with Rethinking) plateau by about three iterations while cost grows linearly. Both run a fixed schedule on every input, and the open problem the file names is the lack of grounded, cheap stopping rules. iRe2f shows that errors in first-pass structures are sparse and can be fixed with cheap local updates, at 152.5 vs 98 sentences/sec. That is the parsing analogue of our heavy-tail hypothesis, but iRe2f still refines every sentence.

Single-pass distillation is a competing answer to latency. Meta-Reflection folds iterative self-correction into a codebook inside a language model. DCFold collapses AlphaFold3's recycling and denoising into one step at 15× speedup, with some accuracy loss (78.6% vs 82.9% on PoseBusters V2). We treat distillation as a direct rival rather than a complement, because the generalization-mismatch results predict that a distilled network inherits the amortization gap on new data.

On the sparse-label side, Point2Mask and Con2R supply the objectives and benchmarks. Both use their iterative or graph machinery only at training time and output a single pass at test time. Neither asks how far that pass sits from the per-image optimum. LabOR shows that targeted extra labels beat uniform labeling, which motivates our secondary use of the decomposition: deciding where to send an annotator.

Research Program

We would build an expander network for each benchmark. Its inputs are the image and its seed channels: rendered points, or the labeled slices. It shares a backbone with a per-image conditioning vector that modulates the decoder, plus a final mask-logit layer. Refinement means gradient steps on $E$ with respect to that conditioning vector and the logits, starting from the network's output, trained end to end through K unrolled steps in the SA-VAE style. Restricting optimization to a low-dimensional code is the stabilizer that the SA-VAE step-size problems call for. Directly optimizing the logits is kept as an ablation.

The reference optimum $m^\ast$ is the best of three candidates under $E$, and can be checked against ground truth: 300 steps from the network's initialization, 300 steps from a seed-only initialization, and the exact Sinkhorn transport solution of Point2Mask where that applies.

Phase one is the decomposition study. For every test image we compute both gaps, in-distribution and under shift. We localize the amortization gap spatially by distance to the nearest seed and by whether a pixel lies between two same-class instances. This directly addresses the open question in the file about how the gap differs between labeled and unlabeled regions. Phase one is informative on its own whatever phase two finds. If the objective gap dominates, the field should invest in priors rather than inference.

Phase two is the gate. Every image gets one refinement step, which serves as a probe. A small regressor then predicts the remaining achievable reduction in $E$ from four cheap features: seed loss after the forward pass, gradient norm at initialization, the drop in $E$ after the probe step, and mask entropy near instance boundaries. Images above a threshold continue for up to K steps. A held-out-seed variant serves as a label-grounded stopping rule: hide 10% of the marks, refine on the rest, and stop when loss on the hidden marks stops falling. This addresses the error-compounding concern, because the stopping signal comes from human labels rather than the model's own outputs.

Phase three uses the decomposition for annotation routing. If an image's predicted amortization gap is small but its predicted error is still high, the objective gap must dominate, and it is flagged for extra clicks rather than extra compute.

Baselines

The naive reference is the trained expander run once, with no refinement. This is the deployment default and the point of zero added latency. Any gated method must beat its quality-at-latency, and we report how much of the per-image-optimum quality it already reaches.

The strongest existing method is per-image optimization of the published objectives. On Pascal VOC 2012 and COCO this is Point2Mask's Sinkhorn transport assignment, run with its released code (LiWentomng/Point2Mask). On CoNSeP and BBBC024 it is the objectness objective. On RETOUCH and BraTS it is Con2R-style regularized fitting per volume. These represent the quality ceiling of each objective.

For calibration, the published standard-protocol numbers are 53.8 PQ for Point2Mask with ResNet-50 on VOC (61.0 with Swin-L), 32.4 PQ on COCO, 48.2 mIoU for Con2R with 24 slices on RETOUCH, 43.2 on BraTS, 0.5546 mIoU on CoNSeP, and 0.6696 IoU on BBBC024. Those protocols supply no seeds at test time. For the deployment-time expansion protocol used here, no reported result was found on Lacuna, so we rerun everything in our protocol.

The third baseline is fixed-K semi-amortized refinement for every image, with K ∈ {1, 3, 10}, following Semi-Amortized Variational Autoencoders. The fourth is the same with trajectory reuse, following BSVI. The fifth is a distilled single-pass expander trained to match refined outputs.

The key ablation replaces the learned gap predictor with the same compute budget spent on randomly chosen images. This isolates whether choosing which images to refine matters.

Fairness rules are fixed in advance. All methods share backbone, training images, seed draws, and GPU. Each gets the same hyperparameter search budget, 20 configurations tuned on a validation split. Latency is measured as wall-clock per image at batch size one. The proposal counts as beating the baselines if gated refinement reaches at least 90% of the full-optimization IoU or PQ gain over single-pass at under 1.5× single-pass mean latency, and lies strictly above the fixed-K curve.

Why not just…?

Why not just gate on prediction confidence? Entropy-based early exit is the cheapest gate and a real threat. The difference is that low confidence flags both kinds of error, and only the amortization part is fixable by compute. A confident but wrong mask produced by a bad prior should not be refined. We run entropy gating at matched compute. If it matches the learned gap predictor, we report that and adopt it, and the remaining contribution is the decomposition and the routing between compute and annotators.

Why not just refine one step everywhere? The diminishing-returns results (Hwang & Park; Learning with Rethinking) suggest most of the gain arrives in the first step. If the gap is heavy-tailed, though, the hard tail needs many steps and the easy majority needs none. The fixed K=1 curve tests this directly. If K=1 already captures 90% of the gain, the gate adds nothing and we say so.

Why not just distill the refinement into the network? DCFold and Meta-Reflection show distillation can remove refinement latency in-distribution. Inference Suboptimality in Variational Autoencoders, however, finds that the amortization gap of a learned encoder grows on unseen data. A distilled expander is still an amortized network, so the prediction is that it closes the in-distribution gap but not the shifted one. We test this with training on VOC and testing on COCO, and training on BBBC024 synthetic cells and testing on real SIM microscopy. If distillation holds up under shift, test-time refinement is not justified.

Why not just scale the network? Cremer et al. find that a more flexible encoder output (flows) shrinks but does not remove the gap, at 43.22 vs 12.86 nats on MNIST. Hwang & Park match heavier U-Nets with 120k weights plus staged refinement. We include a backbone twice as large at matched latency as a single-pass baseline.

Proposed Experiments

The decisive checks are four.

First, the decomposition. Is the amortization gap a substantial share of total expansion error, and is it heavy-tailed? We would call it heavy-tailed if the top 20% of images hold most of the gap. We test this on CoNSeP, BBBC024, VOC points, and RETOUCH or BraTS with 6 to 24 slices. A uniform gap falsifies the gating premise.

Second, gap growth under shift. Does the gap grow under the two transfer pairs above, while the objective gap stays stable?

Third, the quality–latency frontier. We compare gated refinement, entropy gating, fixed-K, BSVI-style trajectory reuse, the distilled expander, and the random-selection ablation. We report area under the IoU-versus-latency curve and the fraction of images refined.

Fourth, annotation routing. Under a fixed time budget, priced at Point2Mask's 22 seconds per image for points, compare sending flagged images to an annotator for extra clicks against uniform extra clicks, in the spirit of LabOR. Success means higher final mask quality per annotation second than uniform clicking.