Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture

Jianwei Zhang, Dong Li, Lituan Wang, Lei ZhangarXiv 2022 · OpenReview

TL;DR

This paper introduces a novel AutoML approach that simultaneously searches for optimal data augmentation policies and neural network architectures for medical image analysis. By addressing the "in-domain sampling bias" in small medical datasets and proposing an "Augmented Density Matching" strategy, their unified search method, USAA, significantly outperforms existing state-of-the-art approaches on the MedMNIST benchmark.

What Problem Does This Paper Address?

Deep learning has become a powerful tool for medical image analysis, but creating effective models often requires extensive human effort and computational resources for tasks like designing neural architectures and selecting data augmentation strategies. Existing AutoML methods often tackle these two crucial components separately, which can be inefficient. Furthermore, small-scale medical image datasets suffer from "in-domain sampling bias," where the training and validation sets are not perfectly representative of each other. This bias can hinder the effectiveness of common auto-augmentation techniques like Density Matching, leading to suboptimal results. The paper aims to bridge this gap by proposing a unified search strategy that accounts for this bias and optimizes both architecture and augmentation concurrently.

Methods

The core of this work lies in two main innovations:

  • Augmented Density Matching: To combat the inefficiency of Density Matching caused by in-domain sampling bias, the authors propose an improved strategy. Instead of directly optimizing for validation performance, they train the model with augmentation policies sampled from a prior distribution. This "Augmented Density Matching" aims to make the augmentation search more robust to the discrepancies between training and validation data.
  • Unified Search (USAA): The paper introduces USAA, an efficient AutoML algorithm that unifies the search for both data augmentation policies and neural architectures. This is achieved by encoding both aspects into a single "individual" within an Evolutionary Algorithm (EA)-based framework. This allows for a simultaneous search, dramatically improving efficiency by finding optimal combinations in a single round. The Area Under the Receiver Operating Characteristics Curve (AUC) is used as the primary optimization objective to handle class imbalance issues prevalent in medical datasets.

Datasets

The primary dataset used for evaluation is MedMNIST [31]. This benchmark is highlighted as a pioneering dataset specifically designed for AutoML in medical image analysis. It comprises 10 diverse sub-datasets covering various medical imaging modalities (histology, X-ray, OCT, etc.) and tasks (binary, multi-class, ordinal regression, multi-label), with resolutions pre-processed to 28x28.

Key Insights and Ideas

  • In-Domain Sampling Bias is Real and Problematic: The paper provides both theoretical and empirical evidence for the inconsistency between train and validation sets in small medical datasets. This bias is shown to directly impact the effectiveness of existing auto-augmentation methods.
  • Augmentation and Architecture are Intertwined: Treating data augmentation and neural architecture search as separate problems is suboptimal. A unified approach can discover synergistic combinations that lead to better performance.
  • Simplicity Can Be Powerful: The authors found that a "Small Search Space" for augmentation (with fewer candidate operations) performed better than a "Large Search Space." This suggests that a more constrained, well-designed search space can prevent overfitting to specific dataset quirks and lead to more generalizable solutions.
  • Optimal Capacity Isn't Just About Data Scale: While neural capacity (represented by cell number) is shown to correlate with dataset size, the paper also suggests an "optimal augmentation capacity" for a dataset, hinting at a new avenue for future research.

Key Results

The proposed USAA algorithm demonstrated superior performance across the MedMNIST benchmark. Specifically:

  • USAA outperformed state-of-the-art AutoML tools like Auto-sklearn, AutoKeras, and Google AutoML Vision in terms of both average AUC and accuracy.
  • Compared to Google AutoML Vision, USAA achieved a 1.2% higher average AUC and 6.3% higher average ACC.
  • USAA also showed significant improvements over other auto-augmentation methods like FastAugment and FasterAugment, demonstrating the effectiveness of the Augmented Density Matching strategy.
  • The unified search approach (USAA) achieved 1.1% higher average AUC and 2% higher average ACC compared to HOPNAS (which only searches neural architecture).

Limitations and Future Work

  • Limitations: The paper acknowledges that the "in-domain sampling bias" is a key challenge for small-scale medical image datasets. While their method addresses it, the inherent nature of such datasets remains a constraint. The search process, while improved, still involves a computational budget.
  • Future Work: The authors suggest that future work could benefit from further constraining the search space for both neural architecture and augmentation policy based on data scale. They also highlight the potential for an "optimal augmentation capacity" for a dataset, which could be a new direction for research. Further exploration into the relationship between data scale, optimal neural capacity, and augmentation strategy is also suggested.

Summary updated

Abstract

Automated data augmentation, which aims at engineering augmentation policy automatically, recently draw a growing research interest. Many previous auto-augmentation methods utilized a Density Matching strategy by evaluating policies in terms of the test-time augmentation performance. In this paper, we theoretically and empirically demonstrated the inconsistency between the train and validation set of small-scale medical image datasets, referred to as in-domain sampling bias. Next, we demonstrated that the in-domain sampling bias might cause the inefficiency of Density Matching. To address the problem, an improved augmentation search strategy, named Augmented Density Matching, was proposed by randomly sampling policies from a prior distribution for training. Moreover, an efficient automatical machine learning(AutoML) algorithm was proposed by unifying the search on data augmentation and neural architecture. Experimental results indicated that the proposed methods outperformed state-of-the-art approaches on MedMNIST, a pioneering benchmark designed for AutoML in medical image analysis.
BibTeX
@misc{zhang2022auto,
  author = {Jianwei Zhang and Dong Li and Lituan Wang and Lei Zhang},
  title = {{Auto Machine Learning for Medical Image Analysis by Unifying the Search on Data Augmentation and Neural Architecture}},
  year = {2022},
  eprint = {2207.10351},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2207.10351},
}
Figure