Proximal Pseudo-Labels for Test-Time Segmentation Adaptation

A segmentation model trained at one hospital often breaks at another. The failures are rarely a uniform loss of accuracy. They look anatomically wrong: stray islands of "organ" in background tissue, ragged boundaries, organs at implausible sizes. Test-time adaptation updates the model on unlabeled images from the new site as they arrive. Its usual objectives, such as making predictions more confident, have no notion of anatomy. They can make the model confidently wrong, and a gain in overlap scores does not reliably carry over to boundary accuracy. We propose to adapt the model by pulling each prediction toward its nearest anatomically plausible version. That version is computed exactly by a proximal operator: a small convex optimization that trades off staying close to the prediction against satisfying non-smooth shape rules, namely total variation (a penalty on boundary length and fragmentation), per-class volume bounds, and valid class probabilities.

The mathematical device is the Moreau envelope, a smoothed version of a non-smooth function whose gradient is the gap between a point and its proximal projection, divided by a smoothing parameter $\lambda$. Applied to the prediction rather than to the network weights, it gives an adaptation loss with a provably well-behaved gradient. The same quantity serves two jobs. It is the pseudo-label the model is trained toward, and its size is a label-free signal of how implausible the prediction is. We use that signal to decide when adapting is safe and when an image should be flagged instead.

Core Claim

For a network output $p_\theta(x)$, a softmax segmentation of image $x$ under weights $\theta$, let $f$ be a convex, non-smooth anatomical penalty. In our case $f(p) = \mu\,\mathrm{TV}(p) + \iota_{\mathcal{C}}(p)$, where $\mathrm{TV}$ is total variation, $\mu$ its weight, and $\iota_{\mathcal{C}}$ is zero when $p$ lies in the set $\mathcal{C}$ of per-pixel probability vectors whose class volumes fall inside ranges estimated from source training masks, and infinite otherwise. The test-time loss is the Moreau envelope

$$ \mathcal{L}(\theta) = e_\lambda f\big(p_\theta(x)\big) = \min_{q}\; f(q) + \tfrac{1}{2\lambda}\|p_\theta(x) - q\|^2 , $$

whose gradient with respect to the prediction is $\big(p_\theta(x) - \mathrm{prox}_{\lambda f}(p_\theta(x))\big)/\lambda$. In practice this is regression toward a stop-gradient target, the proximal mask $q^\star = \mathrm{prox}_{\lambda f}(p_\theta(x))$. We make two claims. First, because $f$ is convex in output space, $q^\star$ is unique and the output-space gradient is exactly $1/\lambda$-Lipschitz. This sidesteps the open problem of Moreau smoothing for non-convex deep losses: we smooth the prior, not the network. Second, the displacement $\|p_\theta(x) - q^\star\|$ is a per-image indicator of shift severity. It can gate updates and detect negative transfer before it compounds. We expect the gains to show up mainly in boundary metrics such as the 95th-percentile Hausdorff distance (HD95), because total variation directly penalizes islands and spurs.

We should be frank that, mechanically, this is pseudo-label self-training with targets cleaned by a convex denoiser. What the Moreau view adds is a principled link between step size and $\lambda$, an exact smooth surrogate in place of an ad hoc relaxation, and the gating signal for free. The central empirical question is whether weight updates driven by these targets beat simply applying the same proximal cleanup to each output. The experiments are designed to settle that.

Relation to Prior Work

The optimization tools come from work that uses Moreau envelopes to handle non-smooth structure. MoreauGrad computes envelope gradients of a classifier score, with $L_1$ and group-sparse penalties, to get stable saliency maps. It never updates weights and never touches medical data. We reuse its core observation, that envelope gradients are stable under perturbation, but apply it to training signals. Moreau-Yoshida variational transport smooths total-variation and $L_1$ regularizers over distributions for generative modeling under Gaussian noise. We borrow the idea of smoothing total variation, but for dense discriminative prediction under real scanner shift. Several works motivate the "prox inside a larger loop" design and the argument that Moreau smoothing beats subgradients for acceleration and stability: ASVRG and SMAG for max-structured weakly convex problems on convergence, and LPGD on differentiating through embedded optimization layers.

The only prior work pairing Moreau envelopes with medical data is FLAME. It anchors personalized federated models to a global one with a Moreau quadratic term and tests on simulated label skew in Medical MNIST classification. It smooths in parameter space during training, while we smooth in output space at deployment on physical scanner shift. MEMLQR likewise uses a parameter-space envelope for meta-learning in low-dimensional control, so it offers no evidence about image-level anatomical priors.

On the medical side, MedSeg-TTA benchmarks 20 adaptation methods across seven modalities. It documents negative transfer and the disagreement between overlap and boundary metrics that motivate us. PTTEA is the closest competitor. It trains a neural energy model as a shape prior on source data and minimizes its output at test time. It needs an extra learned network and adversarial negative mining. Its prior is smooth by construction, so it cannot encode hard volume constraints or exact total variation. ErrorNet corrects vessel segmentations using a learned error model. Both show that anatomical priors help across domains. We test whether a training-free convex prior, handled exactly, gives comparable gains at lower cost. INPUTADAPTER constrains the design in a different way: regulated devices may forbid weight updates. Our method therefore has to degrade gracefully to a frozen-weight mode, which is the output-space proximal step alone.

Research Program

We will build the proximal operator for 2D slices first. It combines isotropic total variation on each class channel with projection onto the probability simplex and onto per-class volume intervals, solved by a GPU primal–dual method with around 20 iterations, warm-started from the previous step's solution. The volume intervals come from percentiles of source training masks, so no target labels are ever used. Adaptation follows the MedSeg-TTA protocol: one pass over the target stream, updating normalization and affine parameters only, as TENT does, with batch size down to 1. Each step computes $q^\star$ and takes a gradient step with learning rate tied to $\lambda$, so that the Lipschitz constant of the output-space gradient stays fixed as $\lambda$ varies.

The gate works as follows. Normalized displacement is logged per image. Updates are skipped when it falls below a low threshold, since the prediction is already plausible. Images above a high threshold are skipped and flagged, since the prediction is too implausible to trust as a pseudo-label. If the running average displacement rises over a window, the adapted parameters are reset to the source weights. Thresholds are set from source validation data only.

The analysis has three parts. We will measure how well displacement predicts per-image Dice loss on the target, using rank correlation and AUROC for detecting Dice below 0.5. We will sweep $\lambda$ to map the tradeoff between fidelity and smoothing, which is the known sensitivity problem of Moreau smoothing. And we will profile the cost of the inner proximal solve against the forward pass.

Baselines

The naive reference is the source-trained UNet applied unchanged, which MedSeg-TTA supplies for every modality pair. Adaptation that falls below it counts as negative transfer, and we will report the fraction of target images where each method ends up worse than this reference.

The strongest existing method on the cardiac, spinal cord and lung-X-ray suites is PTTEA. It reports roughly a 9% Dice improvement over the second-best method on ACDC→LVQuant with a UNet backbone. We will run it from its described procedure on identical backbones, identical target streams and identical updated-parameter subsets, giving both methods the same hyperparameter search budget on source-validation data. On the MedSeg-TTA ultrasound pair, the strongest number in the file is SFDA-FSM, which raised Dice from 0.4811 to 0.8333 on TN3K→DDTI. That is the bar there. Beating PTTEA means matching its Dice within noise while improving HD95, or improving both, across at least the ACDC→LVQuant and M&M shifts, without a trained energy network. We found no reported HD95 figures on Lacuna for PTTEA or MedSeg-TTA methods, so boundary comparisons will rest on our own reruns.

We also include TENT-style entropy minimization, which MedSeg-TTA covers, because it is the default simple adaptation method.

The ablation that isolates the proposal's central question is proximal post-processing alone: apply $\mathrm{prox}_{\lambda f}$ to each frozen-model output and never update weights. Three further ablations isolate the Moreau ingredient. The first uses the same total-variation and volume penalty applied directly as a loss via subgradients. The second uses a common smooth relaxation, a Charbonnier-smoothed total variation plus a quadratic volume penalty. The third runs our method with the gate disabled. All ablations share the same budget for tuning the prior weight.

Why not just…?

Why not just clean up each output with the proximal step and skip adaptation? This is the most serious threat. Much of the boundary gain may come from denoising alone, and this mode also satisfies frozen-weight regulation. Our method can win only through transfer across images: updated normalization statistics should improve image $k+1$ before its own cleanup, and they should fix errors the image-agnostic prior cannot, such as a misplaced but smooth organ. The test is to compare "adapt, then prox" against "prox only" on the same streams, and to check the raw pre-prox Dice of adapted models on held-out target images. If prox-only matches, we will report it as the practical recommendation and treat adaptation as unnecessary on those shifts.

Why not just minimize entropy as TENT does? Entropy has no anatomical content. MedSeg-TTA reports that such output regularizers can raise Dice while HD95 worsens, and can fall below the unadapted model. PTTEA also reports beating TENT across all its datasets. Our negative-transfer rate and HD95 columns measure this directly.

Why not just use PTTEA's learned energy prior? It may well win on Dice where shape varies in ways a convex prior cannot express. It costs a separately trained energy network per task, with adversarial negative mining, and it provides no exact constraint handling or calibrated per-image displacement. If PTTEA wins on Dice, we will test whether our displacement gate added on top of PTTEA reduces its failures. That would still make the Moreau signal useful.

Why not just add a subgradient or smoothed total-variation term to the loss? The subgradient and smooth-relaxation ablations answer this. The optimization literature (ASVRG) predicts slower, noisier progress from subgradients, but a Charbonnier relaxation may perform comparably. If it does, the contribution reduces to the gate and the exact volume constraints, and we will say so.

Why not just train with heavy augmentation? Deep Stacked Transformations cut unseen-domain degradation to 11%, yet AMOS still shows drops of over 20% on MRI from unseen scanners. We will train source models both with and without stacked augmentation and ask whether proximal adaptation still helps on top. The two approaches are complementary rather than competing.

Proposed Experiments

The main evaluation covers all seven MedSeg-TTA modality pairs, using its no-leakage protocol: BraTS-GLI→SSA, LiTS→3D-IRCADB, MESSIDOR→RIGA+, TN3K→DDTI, CRAG→GlaS, ISIC→PH2, and SZ-CXR→Montgomery. We add the PTTEA suites ACDC→LVQuant, M&M vendor shifts, GMSC across four sites, and Shenzhen/Montgomery/JSRT. We report Dice, HD95 and average surface distance, together with the negative-transfer rate, over three seeds.

Three checks are decisive. The first is whether adapt-then-prox beats prox-only on held-out target images. The second is whether the Moreau loss beats its subgradient and Charbonnier ablations on HD95 at equal tuning budget. The third is whether normalized displacement predicts per-image failure with AUROC well above chance, and whether gating lowers the negative-transfer rate without lowering mean Dice.

As a stretch goal, we will extend the operator to 3D total variation on AMOS CT→MRI to test whether the method narrows its reported drop of more than 20%. We will report wall-clock overhead per slice against the forward pass throughout.