Progressive Test Time Energy Adaptation for Medical Image Segmentation

Xiaoran Zhang, Byung-Woo Hong, Hyoungseob Park, Daniel H. Pak, Anne-Marie Rickmann, Lawrence H. Staib, James S. Duncan, Alex Wong

Introduction

In clinical environments, medical image segmentation models frequently encounter "domain shift"—a performance degradation that occurs when a model trained on data from one scanner or protocol is applied to another. While Test-Time Adaptation (TTA) offers a way to update models using only the current patient's data, standard methods often rely on simple statistics like entropy minimization. These methods lack anatomical awareness, frequently leading to unrealistic or "anatomically impossible" segmentation results when the domain gap is significant.

High-level overview of the energy-based adaptation loop.

This paper introduces a Progressive Test Time Energy Adaptation framework. By utilizing a region-based Shape Energy Model, the system identifies erroneous predictions in real-time and iteratively updates the network to produce plausible results without requiring access to the original source training data.

Methodology: The Two-Stage Framework

The proposed approach operates in two distinct phases: source preparation and test-time adaptation.

1. Source Preparation: Training the "Anatomy Teacher"

To teach the system what valid anatomy looks like, a Region-based Energy Model ($g_\phi$) is trained on the source dataset. Since the training set only contains correct examples, the researchers use adversarial perturbations (specifically FGSM) to intentionally generate "broken" predictions. The energy model learns to distinguish between "low-energy" (plausible, in-distribution shapes) and "high-energy" (erroneous, out-of-distribution shapes) at a patch level.

2. Progressive Test-Time Adaptation: The "On-the-fly Fixer"

During inference on a new target image ($I_t$), the energy model remains frozen. The segmentation network ($f_\theta$) generates an initial prediction, which is evaluated by the energy model. If high-energy regions (errors) are detected, the framework calculates a gradient to update the segmentation network's parameters—specifically the BatchNorm layers. This iterative process "steers" the model toward producing anatomically correct masks while maintaining consistency with learned features.

Detailed workflow showing (a) energy model training on source data and (b) progressive adaptation on target data.

Experimental Evaluation

The framework was rigorously validated across eight public datasets covering various modalities and anatomies, including Cardiac MRI (ACDC, LVQuant, MyoPS, M&M), Spinal Cord MRI (GMSC), and Lung X-rays (CHN, MCU, JSRT). The approach proved architecture-agnostic, demonstrating effectiveness with standard UNets, MedNeXt, and transformer-based SwinUNETR backbones.

Key Results

  • Accuracy Gains: On the ACDC $\rightarrow$ LVQuant transfer task, the method improved the Dice Similarity Coefficient (DSC) by approximately 9% for the Left Ventricle and 5% for the Myocardium compared to the next best baseline.
  • Robust Error Detection: The energy model achieved over 92% accuracy in identifying localized segmentation failures across different backbones.
  • Computational Efficiency: The system is significantly faster than generative energy-based methods, achieving a 4.5x speedup over CoTTA and a 1.3x speedup over TEA.

Discussion and Future Directions

A primary advantage of this method is its region-based approach, which allows the network to pinpoint exactly where a segmentation is failing (e.g., a jagged border) and apply localized corrections. By updating model weights rather than just the output mask, the system avoids the instability often seen in other TTA methods.

However, the framework currently assumes a covariate shift where the underlying anatomy remains similar. It is also currently limited to 2D slices; extending this to 3D volumetric data is a necessary next step for broader clinical utility. Despite these limitations, the progressive energy-based update provides a robust and efficient path for real-time clinical model adaptation.

Summary updated

BibTeX
@inproceedings{zhang2025progressive,
  author = {Xiaoran Zhang and Byung-Woo Hong and Hyoungseob Park and Daniel H. Pak and Anne-Marie Rickmann and Lawrence H. Staib and James S. Duncan and Alex Wong},
  title = {{Progressive Test Time Energy Adaptation for Medical Image Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure