Searching Learning Strategy with Reinforcement Learning for 3D Medical Image Segmentation

Dong Yang, Holger Roth, Ziyue Xu, Fausto Milletari, Ling Zhang, Daguang Xu

Abstract

Deep neural network (DNN) based approaches have been widely investigated and deployed in medical image analysis. For example, fully convolutional neural networks (FCN) achieve the state-of-the-art performance in several applications of 2D/3D medical image segmentation. Even the baseline neural network models (U-Net, V-Net, etc.) have been proven to be very effective and efficient when the training process is set up properly. Nevertheless, to fully exploit the potentials of neural networks, we propose an automated searching approach for the optimal training strategy with reinforcement learning. The proposed approach can be utilized for tuning hyper-parameters, and selecting necessary data augmentation with certain probabilities. The proposed approach is validated on several tasks of 3D medical image segmentation. The performance of the baseline model is boosted after searching, and it can achieve comparable accuracy to other manually-tuned state-of-the-art segmentation approaches.

Introduction

While baseline deep learning architectures like U-Net and V-Net are highly effective for 3D medical image segmentation, their peak performance is often bottlenecked by the training strategy. Engineers typically rely on domain expertise, heuristics, or computationally expensive grid searches to set learning rates and design data augmentation pipelines.

Furthermore, because 3D medical imaging is highly variable (e.g., CT vs. MRI) and computationally massive, standard automated techniques like Neural Architecture Search (NAS) are often too expensive to run. This research addresses the need for an automated, efficient way to find the best hyperparameter and augmentation configurations for 3D medical images without relying on human guesswork.

Methodology

The authors frame hyperparameter and augmentation tuning as a Reinforcement Learning (RL) problem, adapting the "AutoAugment" concept for the heavy compute requirements of 3D medical imaging.

Search Space and Controller

Instead of discrete choices, the framework defines a continuous search space mapped to [0,1]. This space includes the learning rate and the probability of applying specific augmentations such as sharpening, smoothing, Gaussian noise, contrast adjustment, and random intensity shifts.

The RL framework uses an RNN controller to propose training strategies, receiving validation accuracy as a reward signal.

The "controller" is a Recurrent Neural Network (RNN) trained via Proximal Policy Optimization (PPO). As shown in Figure 1, the RNN generates a training strategy (the "action"). A 3D segmentation model is then trained using this strategy, and the resulting validation accuracy serves as the "reward." The RNN uses this reward to update its weights, allowing it to propose more effective strategies in subsequent iterations.

Experimental Results

The approach was evaluated on four tasks from the Medical Segmentation Decathlon (MSD) challenge: Left Atrium (MRI), Lung Tumor (CT), Pancreas and Tumor (CT), and Spleen (CT).

Performance Gains

The proposed RL search consistently outperformed baseline models and traditional search methods. In fine-tuning scenarios, the RL approach achieved a Dice score of 0.92 on Task02 (MRI) and Task09 (CT), surpassing both "No Augmentation" and hill-climbing baselines.

Comparison of the proposed approach against routine training strategies across three MSD tasks.

When applying the RL-discovered strategy to train models from scratch, it significantly outperformed a "routine" strategy (which uses a fixed 0.5 probability for augmentations). As seen in Figure 3, the Dice score for Lung CT (Task06) jumped from 0.383 to 0.449, and Pancreas CT (Task07) improved from 0.491 to 0.519.

Discussion and Conclusion

A key insight from the study is that imaging modality dictates the optimal augmentation strategy. The RL agent discovered that CT scans benefit heavily from random intensity scale shifts, while MRI scans prefer image sharpening to strengthen region-of-interest boundaries. Additionally, the authors demonstrated that running the RL search on a pre-trained model (fine-tuning for 200 epochs) serves as an effective shortcut to find strategies that work when training from scratch.

While the approach is more efficient than full 3D NAS, it remains computationally intensive, requiring approximately 48 hours on 50 NVIDIA V100 GPUs for a full search. Future work may involve extending the reward function to include multi-dimensional objectives, such as balancing accuracy with inference speed or anatomical constraints.

Summary updated

Abstract

Deep neural network (DNN) based approaches have been widely investigated and deployed in medical image analysis. For example, fully convolutional neural networks (FCN) achieve the state-of-the-art performance in several applications of 2D/3D medical image segmentation. Even the baseline neural network models (U-Net, V-Net, etc.) have been proven to be very effective and efficient when the training process is set up properly. Nevertheless, to fully exploit the potentials of neural networks, we propose an automated searching approach for the optimal training strategy with reinforcement learning. The proposed approach can be utilized for tuning hyper-parameters, and selecting necessary data augmentation with certain probabilities. The proposed approach is validated on several tasks of 3D medical image segmentation. The performance of the baseline model is boosted after searching, and it can achieve comparable accuracy to other manually-tuned state-of-the-art segmentation approaches.
BibTeX
@inproceedings{yang2019searching,
  author = {Dong Yang and Holger Roth and Ziyue Xu and Fausto Milletari and Ling Zhang and Daguang Xu},
  title = {{Searching Learning Strategy with Reinforcement Learning for 3D Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32245-8\_1},
}
Figure