Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation in Medical Image Segmentation

Shiyu Liu, Daoqiang Zhang, Xiaoke HaoAAAI 2025 · DOI

TL;DR

This paper introduces an "Efficient Deformable Convolutional Prompt" (EDCP) to tackle the challenge of adapting medical image segmentation models to new, unseen data distributions during testing (Continual Test-Time Adaptation or CTTA). EDCP achieves this by using a lightweight, deformable convolutional prompt that learns to adjust the input image, significantly reducing computational overhead and improving adaptation efficiency compared to existing methods.

What Problem Does This Paper Address?

Medical image segmentation models perform exceptionally well when the training and testing data share the same characteristics. However, in real-world clinical settings, this is rarely the case. Differences in imaging protocols, scanner types, and even patient populations create a "domain gap," leading to a significant drop in model performance. Continual Test-Time Adaptation (CTTA) aims to address this by allowing a pre-trained model to adapt to new, unlabeled data distributions on the fly during inference, without needing access to the original training data. Existing CTTA methods often rely on complex prompt injection strategies or update large parts of the model, leading to high computational costs and parameter bloat, especially for high-resolution medical images. This paper tackles the need for an efficient and effective CTTA method that can adapt to continually changing domains with minimal computational burden.

Methods

The core innovation is the Efficient Deformable Convolutional Prompt (EDCP). Instead of traditional prompts that might be added at the same scale as the input or require complex selection strategies, EDCP leverages the inherent "inductive bias" of convolutions.

Here's a breakdown of the key techniques:

  • Deformable Convolutional Prompt: EDCP uses a convolutional prompt that is made "deformable." This means it can learn to adjust sampling locations within the convolution kernel, allowing it to adapt to fine-grained shifts in image features and structures at the pixel level. This is achieved by adding an "offset branch" that predicts these adjustments.
  • Transformation Decomposition: To manage the parameter imbalance between the convolution and offset branches (the offset branch can be much larger), the authors decompose the offset transformation into two parts. The second part, represented by a transformation matrix, is stored in an "offset bank."
  • Offset Bank as Domain Indicator: This offset bank acts as an implicit domain indicator. By comparing the transformation matrix of the current input with those already in the bank, the system can identify if the current data is similar to previously seen domains.
  • Source-Target Alignment: The adaptation process involves two alignment strategies:
  • Input-level Alignment: Encourages structural consistency between an augmented version of the input and the adapted original input.
  • Layer-wise Alignment: Measures differences in statistics within Batch Normalization (BN) layers between the target and source domains, guiding prompt updates without unfreezing the source model's BN layers.
  • Fast Inference: This is a crucial efficiency component. If the similarity between the current input's transformation matrix and those in the offset bank exceeds a threshold, the system skips the training/adaptation phase for that sample and proceeds directly to inference. This significantly speeds up the process by avoiding unnecessary updates on similar data.

Datasets

The paper evaluates EDCP on several medical imaging datasets across different domain shift scenarios:

  • OD/OC Segmentation Dataset: A collection of five public fundus datasets from different medical centers, representing cross-site domain shifts.
  • Brain Tumor Segmentation (BraTS 2020) Dataset: A multimodal dataset with four MRI sequences (T1, T1ce, T2, FLAIR), used to evaluate cross-sequence adaptation.
  • Cardiac Segmentation Dataset: A dataset with CT and MRI scans of cardiac regions, used for cross-modality adaptation.

No new benchmarks were introduced; existing, well-established datasets were used to demonstrate the method's effectiveness.

Key Insights and Ideas

  • Deformable Convolution is Key for Fine-Grained Adaptation: Simply using a convolutional prompt isn't enough for significant domain shifts. Making it deformable allows it to adapt to structural changes at the pixel level, which is crucial for medical images where anatomical structures can shift.
  • Offset Bank as an Implicit Domain Indicator: Storing transformation matrices in an offset bank is a clever way to quickly assess domain similarity. This allows for intelligent skipping of training steps, a major contributor to efficiency.
  • Parameter Balance is Crucial for Efficiency: The decomposition of the offset transformation is vital. Without it, the offset branch would dominate the parameter count, negating the "efficient" aspect of the prompt.
  • Input and Layer-wise Alignment is More Efficient than Output Alignment: Aligning statistics at the input and BN layer levels is faster than waiting for a full forward pass to align output predictions, especially important for real-time CTTA.
  • Fast Inference is a Game-Changer for CTTA: The ability to skip training on similar samples dramatically reduces computational cost and training time, making CTTA practical for real-world deployment.

Key Results

  • Superior Performance: EDCP consistently outperforms state-of-the-art methods across all tested scenarios (cross-site, cross-sequence, cross-modality) for both 2D and 3D medical image segmentation tasks. For instance, in the OD/OC segmentation task, EDCP achieved the highest Dice Score (DSC) of 69.03%.
  • Efficiency Gains: EDCP significantly reduces training time and computational overhead. It achieves an average of only 0.03 backward passes per sample during adaptation due to the fast inference mechanism, compared to other methods that require multiple backward passes.
  • Parameter Efficiency: EDCP uses a remarkably small number of trainable parameters (186 in the ablation study) compared to methods that update BN layers (e.g., TENT with 21184 parameters).
  • Robustness to Long-Term Adaptation: In experiments simulating long-term continual adaptation, EDCP showed minimal performance degradation and stabilized quickly, demonstrating resilience against catastrophic forgetting.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while EDCP significantly reduces training time, it still requires some adaptation steps.
  • The "similarity threshold" ($\gamma$) for fast inference is a hyperparameter that might need tuning.
  • Future Work:
    • The authors suggest that their framework is generalizable and could be applied to other domains beyond medical imaging.
  • Further exploration of the offset bank's role and potential for more sophisticated domain detection could be beneficial.
  • Investigating adaptive strategies for the similarity threshold $\gamma$ could further optimize the fast inference mechanism.

Summary updated

BibTeX
@inproceedings{liu2025efficient,
  author = {Shiyu Liu and Daoqiang Zhang and Xiaoke Hao},
  title = {{Efficient Deformable Convolutional Prompt for Continual Test-Time Adaptation in Medical Image Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32591},
}
Figure