On-the-Fly Test-time Adaptation for Medical Image Segmentation

Jeya Maria Jose Valanarasu, Pengfei Guo, Vibashan Vs, Vishal M. Patel

Abstract

One major problem in deep learning-based solutions for medical imaging is the drop in performance when a model is tested on a data distribution different from the one that it is trained on. Adapting the source model to target data distribution at test-time is an efficient solution for the data-shift problem. Previous methods solve this by adapting the model to target distribution by using techniques like entropy minimization or regularization. In these methods, the models are still updated by back-propagation using an unsupervised loss on complete test data distribution. In real-world clinical settings, it makes more sense to adapt a model to a new test image on-the-fly and avoid model update during inference due to privacy concerns and lack of computing resource at deployment. To this end, we propose a new setting - On-the-Fly Adaptation which is zero-shot and episodic (i.e., the model is adapted to a single image at a time and also does not perform any back-propagation during test-time). To achieve this, we propose a new framework called Adaptive UNet where each convolutional block is equipped with an adaptive normalization layer to adapt the features with respect to a domain code. The domain code is generated using a pre-trained encoder trained on a large corpus of medical images. During test-time, the model takes in just the new test image and generates a domain code to adapt the features of source model according to the test data. We validate the performance on both 2D and 3D data distribution shifts where we get a better performance compared to previous test-time adaptation methods.

Introduction

Deep learning models for medical image segmentation frequently encounter performance degradation due to domain shift. This occurs when a model trained on data from one source (e.g., a specific hospital or scanner) is deployed on data from another with different parameters or demographics.

Existing solutions, known as Test-Time Adaptation (TTA), typically mitigate this by updating model weights using unsupervised losses on the test data. However, these approaches often require backpropagation during inference, necessitating significant computational resources and potentially compromising patient privacy. Furthermore, they typically rely on batches of test data, which is impractical in clinical settings where data often arrives episodically—one patient at a time.

On-the-Fly Adaptation Setting

To address these limitations, we introduce the On-the-Fly Adaptation setting. This approach is zero-shot and episodic, meaning the model adapts to a single test image instantaneously without any backpropagation or memory of previous samples. This contrasts sharply with "Fully test-time adaptation," which requires iterative updates and access to the broader test distribution.

Comparison of (a) Fully test-time adaptation and (b) the proposed On-the-fly adaptation.

Methods: The Adaptive UNet

The proposed framework, Adaptive UNet, achieves on-the-fly adaptation through a feed-forward mechanism. It consists of two primary components:

  • Domain Prior Generator (DPG): A pre-trained encoder that extracts a "Domain Code" from the input image. This code captures the specific "style" or domain characteristics of the image (e.g., scanner noise, contrast levels).
  • Adaptive Normalization Layers: The standard normalization layers in the UNet are replaced with adaptive versions. These layers use the Domain Code from the DPG to scale and shift internal feature maps, effectively "normalizing" the features to a space the pre-trained segmentation network can process accurately.
The Adaptive UNet architecture featuring the Domain Prior Generator (DPG) and Adaptive Normalization layers.

Experimental Results

We evaluated Adaptive UNet on both 2D retinal vessel segmentation (CHASE, RITE, HRF) and 3D brain tumor segmentation (BraTS 2019) tasks.

  • 2D Performance: In shifts between retinal datasets, our method consistently outperformed direct testing and standard TTA methods like TENT. For instance, in a shift from CHASE to HRF, Adaptive UNet achieved a Dice score of 70.19, compared to 61.95 for direct testing and 61.20 for TENT.
  • 3D Performance: In brain tumor segmentation, the model demonstrated robustness to cross-modality shifts (e.g., T1 to T1ce). While cross-modality adaptation remains challenging, Adaptive UNet showed significant gains in segmenting the "Whole Tumor" and "Tumor Core" compared to one-shot backpropagation-based methods.

Conclusion

Adaptive UNet provides a computationally efficient and privacy-preserving solution for domain shift in medical imaging. By enabling on-the-fly adaptation through a simple feed-forward pass, it facilitates the deployment of robust segmentation models in resource-constrained clinical environments. Future work will focus on enhancing the Domain Prior Generator to better handle aggressive cross-modality shifts.

Summary updated

Abstract

One major problem in deep learning-based solutions for medical imaging is the drop in performance when a model is tested on a data distribution different from the one that it is trained on. Adapting the source model to target data distribution at test-time is an efficient solution for the data-shift problem. Previous methods solve this by adapting the model to target distribution by using techniques like entropy minimization or regularization. In these methods, the models are still updated by back-propagation using an unsupervised loss on complete test data distribution. In real-world clinical settings, it makes more sense to adapt a model to a new test image on-the-fly and avoid model update during inference due to privacy concerns and lack of computing resource at deployment. To this end, we propose a new setting - On-the-Fly Adaptation which is zero-shot and episodic (i.e., the model is adapted to a single image at a time and also does not perform any back-propagation during test-time). To achieve this, we propose a new framework called Adaptive UNet where each convolutional block is equipped with an adaptive batch normalization layer to adapt the features with respect to a domain code. The domain code is generated using a pre-trained encoder trained on a large corpus of medical images. During test-time, the model takes in just the new test image and generates a domain code to adapt the features of source model according to the test data. We validate the performance on both 2D and 3D data distribution shifts where we get a better performance compared to previous test-time adaptation methods. Code is available at https://github.com/jeya-maria-jose/On-The-Fly-Adaptation
BibTeX
@inproceedings{valanarasu2023fly,
  author = {Jeya Maria Jose Valanarasu and Pengfei Guo and Vibashan VS and Vishal M. Patel},
  title = {{On-the-Fly Test-time Adaptation for Medical Image Segmentation}},
  booktitle = {MIDL},
  year = {2023},
  url = {https://openreview.net/forum?id=UQDalTzrEg},
}
Figure