Searching Learning Strategy with Reinforcement Learning for 3D Medical Image Segmentation

Dong Yang, Holger Roth, Ziyue Xu, Fausto Milletari, Ling Zhang, Daguang Xu

TL;DR

Training 3D medical image segmentation models requires tedious manual tuning of hyperparameters and data augmentation pipelines. This paper introduces a Reinforcement Learning (RL) framework to automatically search for the optimal training strategy, treating validation accuracy as the reward. The automated approach significantly boosts baseline model performance, proving that how you train a network is just as important as the architecture itself.

What Problem Does This Paper Address?

While baseline deep learning architectures like U-Net and V-Net are highly effective for 3D medical image segmentation, their peak performance is heavily bottlenecked by the training strategy. Engineers typically rely on domain expertise, heuristics, or computationally expensive grid searches to set learning rates and design data augmentation pipelines.

Furthermore, because 3D medical imaging is highly variable (e.g., CT vs. MRI) and computationally massive, standard automated techniques like Neural Architecture Search (NAS) are often too expensive to run. The authors tackle the need for an automated, efficient way to find the best hyperparameter and augmentation configurations for 3D medical images without relying on human guesswork.

Methods

The authors frame hyperparameter and augmentation tuning as a Reinforcement Learning problem, expanding on the core ideas of Google's "AutoAugment" but adapting them for the heavy compute requirements of 3D medical imaging.

  • Continuous Search Space: Instead of discrete choices, they define a continuous search space mapped to [0,1]. This space includes the learning rate and the probability of applying specific augmentations (sharpening, smoothing, Gaussian noise, contrast adjustment, and random intensity shifts).
  • The RL Controller: They use a Recurrent Neural Network (RNN) trained via Proximal Policy Optimization (PPO) as the "controller."
  • The Feedback Loop: The RNN generates a training strategy (the "action"). A 3D segmentation model is then trained using this strategy. The resulting validation accuracy serves as the "reward." The RNN uses this reward to update its weights via gradient back-propagation, allowing it to propose better, more tailored strategies in the next iteration.

Datasets

The approach was evaluated on four distinct tasks from the Medical Segmentation Decathlon (MSD) challenge, covering both CT and MRI modalities:

  • Task02: Left Atrium segmentation (MRI)
  • Task06: Lung Tumor segmentation (CT)
  • Task07: Pancreas and Tumor segmentation (CT)
  • Task09: Spleen segmentation (CT)

Key Insights and Ideas

  • Modality dictates augmentation: The RL agent discovered that different imaging modalities naturally prefer different augmentations. For example, CT scans benefited heavily from random intensity scale shifts, while MRI scans preferred image sharpening. The authors hypothesize this is because MRI quality varies wildly, and sharpening helps strengthen region-of-interest boundaries.
  • Continuous probabilities over binary choices: Searching in a continuous probability space for augmentations allowed the model to learn the relative importance of different techniques for a specific dataset, rather than just treating them as binary on/off states.
  • Fine-tuning as a search shortcut: To save massive GPU hours, the authors showed that you can run the RL search on a pre-trained model (fine-tuning it for just 200 epochs) to find a great strategy. This discovered strategy can then be successfully applied to train a fresh model from scratch.

Key Results

The proposed RL search consistently outperformed baseline models and traditional search methods (like Discrete and Continuous Hill Climbing).

  • Fine-tuning scenario: On Task02 (MRI), the RL approach achieved a Dice score of 0.92, compared to 0.88 with no augmentation and 0.90 with discrete hill climbing. On Task09 (CT), it hit 0.92 vs. 0.87 (no augmentation).
  • Training from scratch: When applying the RL-discovered strategy to train models from scratch, it beat a "routine" (fixed 0.5 probability for augmentations) strategy across the board.
  • Task06 (Lung CT): Dice score jumped from 0.383 to 0.449.
  • Task07 (Pancreas CT): Improved from 0.491 to 0.519.
  • Task09 (Spleen CT): Improved from 0.957 to 0.960.

Limitations and Future Work

  • Limitations: While much more efficient than full 3D Neural Architecture Search, the RL approach is still computationally heavy. The full search procedure for training from scratch took about 48 hours utilizing 50 NVIDIA V100 GPUs.
  • Future Work: The authors suggest extending the current single-value reward function (validation accuracy) to a multi-dimensional reward function. This could allow the RL agent to optimize for multiple objectives simultaneously, such as balancing high accuracy with faster inference times or specific anatomical constraints.

Summary updated

Abstract

Deep neural network (DNN) based approaches have been widely investigated and deployed in medical image analysis. For example, fully convolutional neural networks (FCN) achieve the state-of-the-art performance in several applications of 2D/3D medical image segmentation. Even the baseline neural network models (U-Net, V-Net, etc.) have been proven to be very effective and efficient when the training process is set up properly. Nevertheless, to fully exploit the potentials of neural networks, we propose an automated searching approach for the optimal training strategy with reinforcement learning. The proposed approach can be utilized for tuning hyper-parameters, and selecting necessary data augmentation with certain probabilities. The proposed approach is validated on several tasks of 3D medical image segmentation. The performance of the baseline model is boosted after searching, and it can achieve comparable accuracy to other manually-tuned state-of-the-art segmentation approaches.
BibTeX
@inproceedings{yang2019searching,
  author = {Dong Yang and Holger Roth and Ziyue Xu and Fausto Milletari and Ling Zhang and Daguang Xu},
  title = {{Searching Learning Strategy with Reinforcement Learning for 3D Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  doi = {10.1007/978-3-030-32245-8\_1},
}

Loading related content...

Figure