A segmentation model trained on one hospital's scanners often loses a fifth or more of its accuracy on another hospital's scanners. At deployment there are no labels, and privacy rules usually forbid keeping the training scans. One appealing fix is to bring a learned judge along: a network that looks at an image and a candidate mask and predicts how good the mask is. The segmenter can then be nudged at test time toward masks the judge scores highly. The trouble is that the judge is itself out of distribution on the new scanner. A judge that over-rates unfamiliar inputs will reward the segmenter for exploiting its own mistakes. Offline reinforcement learning (RL) has studied this exact problem for years under another name: overestimating the value of actions never seen in the data. Its most reliable cure is strikingly simple. Adding layer normalization (LN), which rescales each example's activations to zero mean and unit variance, to the value network bounds its predictions and keeps updates from spilling over onto unfamiliar inputs.
We propose to treat source-free, test-time adaptation of an RL segmentation agent as an offline-RL problem. Unseen-scanner images play the role of out-of-distribution states, and the source-trained policy plays the role of the behavior policy. We then test whether critic normalization is what makes "maximize the learned judge" adaptation safe rather than self-destructive. The claim is narrow and falsifiable. Removing LN from the critic should cause measurable value overestimation on target scanners, and gradient ascent against that critic should then lower true Dice. With LN, the same procedure should recover a meaningful fraction of the cross-scanner gap without labels or source data, using only a forward model and a handful of gradient steps.
Core Claim
In RL, the critic (or value network) predicts the future reward the agent will collect from a given state and action. In actor-critic segmentation, the state is the image plus the current mask, the action is a per-pixel edit, and the reward measures improvement against ground truth. Once trained on source labels, the critic is a label-free estimate of how much an edit will help. Our central hypothesis is that this estimate stays trustworthy enough to optimize against on unseen scanners only when the critic is layer-normalized.
The mechanism comes from RL theory. Yue et al. show that ordinary networks have a "self-exciting" kernel structure. Their neural tangent kernel describes how a gradient step on one input changes the output on another, and under self-excitation an update at one input inflates values at distant, unseen inputs. LN localizes this kernel. Their eigenvalue measure, SEEM, flags divergence before it happens. Separately, Ball et al. document that LN bounds predicted values for out-of-distribution actions. A different scanner is, to the critic, an out-of-distribution state. We expect the same localization to decide whether test-time optimization against the critic helps or hurts.
Research Program
The agent follows the pixel-level actor-critic design of Liu et al.. That work, PixelDRL-MG, treats each pixel as an agent that iteratively edits a mask over ten steps using a shared feature extractor, self-attention, and separate policy and value heads. It reports strong boundary accuracy and few-shot robustness, but it uses no critic normalization and runs no cross-scanner evaluation. We keep its architecture so that our only interventions are the normalization in the critic and the test-time procedure.
We change one design choice in the reward. Our reward is the per-step change in Dice plus boundary IoU, rather than a pixel-wise match. This makes the critic's output directly interpretable as predicted mask-quality gain.
Training happens on source centers with labels and produces four critic variants that are otherwise identical. The first has no normalization. The second adds LN after each hidden layer. The third uses batch normalization, following the on-robot critic of Bohlinger et al.. The fourth places LN after dropout, following DroQ (Hiraoka et al.), which gives cheap uncertainty estimates without destabilizing value learning.
At test time, for each unseen-scanner volume processed slice by slice, the critic is frozen. We take a small number of gradient steps on the policy head's normalization scales and offsets only. The objective is
$$\max_{\phi}\ \mathbb{E}_{s \sim \text{target}}\big[\,Q_\theta(s, \pi_\phi(s))\,\big] - \beta\, \mathrm{KL}\big(\pi_\phi(\cdot\mid s)\,\|\,\pi_{0}(\cdot\mid s)\big).$$
Here $Q_\theta$ is the frozen critic, $\pi_\phi$ is the adapted policy with tunable parameters $\phi$, $\pi_0$ is the source policy, and $\beta$ sets how far adaptation may drift. The penalty term is the behavior-regularization idea from TD3+BC and ReBRAC (Tarasov et al.), carried over to a test-time setting.
For the DroQ variant, we replace $Q_\theta$ with its dropout mean minus one standard deviation, which is a pessimistic score. We also use the per-volume critic uncertainty as an abstention signal.
The analysis has two parts, and it is as important as the accuracy numbers. First, using held-out labels for evaluation only, we measure each critic's overestimation gap on target domains: predicted quality gain minus realized Dice gain. Second, we compute empirical cross-domain kernel entries, meaning the dot product of the critic's parameter gradients at a source state and a target state. This tells us whether LN actually localizes interference across scanners, as Yue et al. predict for control. We also track a SEEM-style largest-eigenvalue statistic during adaptation, as an early warning of when adaptation will go wrong.
This differs from the closest work in three ways. Tomar et al. adapt inputs using an entropy objective and stored batch-norm statistics, with no learned judge. Li et al. use a "feature-critic" that is a meta-learned training loss, not a value estimate that is optimized against at deployment. Mo et al. show that a boundary-tracing policy transfers across scanners better than static CNNs, but they train it by imitation, with no critic.
Baselines
The naive reference point is the unadapted source agent applied directly to target scanners. Every adaptation method must beat it. Because no prior work reports cross-scanner numbers for PixelDRL-MG, we run it ourselves: no reported result found on Lacuna for PixelDRL-MG under scanner shift.
The second reference is a standard supervised segmenter: nnU-Net or a 3D U-Net, which Ji et al. report at 89.04% mDice on AMOS CT. It is trained with the same augmentations and evaluated on the same splits. This tells us whether the RL agent's starting point is competitive at all.
The strongest existing competitor is OptTTA (Tomar et al.). It is source-free, targets exactly our cross-vendor MRI setting, and reports a 22.5% average Dice improvement over the unadapted model on M&Ms and 9.2% on spinal cord grey matter. We run its released procedure on our backbone and on its own U-Net. To keep the comparison fair, we match the number of test-time gradient steps and wall-clock time per volume, and we give both methods the same hyperparameter budget, tuned on one held-out source center rather than on target labels.
The key ablation removes LN from the critic while keeping everything else fixed. If critic normalization is not what separates helpful from harmful adaptation, the claim fails.
A second ablation trains the critic with Monte Carlo returns instead of bootstrapped TD targets. This isolates whether Bellman training matters at all, or whether a normalized quality regressor suffices.
Beating OptTTA means a higher target-domain Dice on M&Ms and SCGM at equal or lower per-volume compute, with no increase in Betti-number topological error. A smaller gain at markedly lower latency would be reported as a trade-off, not a win.
Why not just…?
Why not just put layer or instance normalization into a supervised U-Net and skip RL? This is the most serious threat. Normalization inside the segmenter may itself absorb intensity shift. However, that segmenter has no judge to optimize against at test time, so it can only be robust passively. We include LN and instance-norm variants of nnU-Net, both unadapted and with OptTTA. If they match our adapted agent, the honest conclusion is that the gain came from segmenter normalization, and the critic adds nothing.
Why not just train a Dice-regression quality network and do gradient ascent on the mask, with no Bellman machinery? This may well work as well as our method, which is why we test it directly. The Monte Carlo critic ablation, plus a standalone LN quality regressor, both sit on the same policy. Our core claim is about normalization of the judge, not about TD learning. If the regressor with LN equals the TD critic with LN, we will report that the offline-RL lens explains the failure mode but bootstrapping is unnecessary. If the regressor without LN collapses in the same way the unnormalized critic does, that still supports the central hypothesis.
Why not just use entropy minimization or batch-norm statistic matching, as OptTTA does? These objectives are cheap and proven. But entropy rewards confident masks, not correct ones, and FeTA shows that fine structures such as grey matter remain the weak point on unseen centers (FeTA 2022). A judge trained on boundary-aware rewards could target exactly those errors. The head-to-head comparison on FeTA grey matter Dice and Betti error settles this.
Why not batch normalization in the critic? The test-time unit is a single patient's slices, so batch statistics are unreliable. BN is still included as a variant, so this is decided by data rather than assumption.
Proposed Experiments
We use four shift settings. M&Ms cardiac MRI and SCGM follow OptTTA's protocol. FeTA 2022 trains on Zurich and Vienna and tests on the unseen CHUV and UCSF centers; for context, the winning team's global average DSC across all test centers was 0.816, with grey matter near 0.75. AMOS trains on MRI from a subset of scanners and tests on held-out scanners, where Ji et al. report drops above 20% mDice. As a small cross-check, we train on Sunnybrook and test on STACOM 2011, where the imitation-trained DPM reaches 0.74 Dice zero-shot (Mo et al.).
Metrics are Dice, 95th-percentile Hausdorff distance, boundary IoU, and Betti-number error. We also report the critic's overestimation gap on target data and the calibration of DroQ uncertainty against realized per-volume error.
The decisive check is a two-by-two result. Without LN, the critic should show a positive overestimation gap on target scanners and adaptation should lower true Dice. With LN, the gap should shrink and adaptation should raise Dice. Cross-domain kernel entries should be smaller under LN in a way that tracks the per-dataset gap. If unnormalized adaptation does not hurt, the offline-RL framing is wrong for this problem. If normalized adaptation does not help beyond OptTTA and LN-nnU-Net, the proposal fails on practical grounds, and we will say so.
A secondary check tests whether the SEEM-style eigenvalue statistic, computed on unlabeled target slices before adaptation, predicts which volumes adaptation will harm. Such a predictor would give a label-free safety gate for deployment.
References
- Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RL
- Efficient Online Reinforcement Learning with Offline Data
- Pixel level deep reinforcement learning for accurate and robust medical image segmentation
- Gait in Eight: Efficient On-Robot Learning for Omnidirectional Quadruped Locomotion
- Dropout Q-Functions for Doubly Efficient Reinforcement Learning
- Revisiting the Minimalist Approach to Offline Reinforcement Learning
- OptTTA: Learnable Test-Time Augmentation for Source-Free Medical Image Segmentation Under Domain Shift
- Feature-Critic Networks for Heterogeneous Domain Generalization
- The Deep Poincaré Map: A Novel Approach for Left Ventricle Segmentation
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Multi-Center Fetal Brain Tissue Annotation (FeTA) Challenge 2022 Results