Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology

Heather D. Couture, J. S. Marron, Charles M. Perou, Melissa A. Troester, Marc Niethammer

Abstract

Multiple instance (MI) learning with a convolutional neural network enables end-to-end training in the presence of weak image-level labels. We propose a new method for aggregating predictions from smaller regions of the image into an image-level classification by using the quantile function. The quantile function provides a more complete description of the heterogeneity within each image, improving image-level classification. We also adapt image augmentation to the MI framework by randomly selecting cropped regions on which to apply MI aggregation during each epoch of training. We validate our method on five different classification tasks for breast tumor histology and provide a visualization method for interpreting local image classifications.

Introduction

Digital pathology images are characterized by their immense scale and high degree of spatial heterogeneity. A single slide can contain a complex mixture of tumor cells, stroma, and immune infiltrates. Despite this complexity, clinical labels are often provided only at the global level (e.g., a single diagnosis for the entire specimen), presenting a "weak label" problem. Standard Single Instance Learning (SIL) approaches, which assign the global label to every small image patch, often fail due to the high noise introduced by non-diagnostic regions. Conversely, traditional Multiple Instance Learning (MIL) methods using simple max or mean pooling often overlook critical local variations. This paper presents a novel MIL framework designed to leverage tissue heterogeneity as a diagnostic signal rather than a source of noise.

Methodology

The proposed framework integrates a Fully Convolutional Network (FCN) with a novel aggregation layer based on quantile functions.

Quantile Aggregation

Instead of relying on a single summary statistic like the maximum or mean, the model aggregates instance-level predictions by sorting them and extracting specific quantiles (e.g., the 10th through 90th percentiles). This quantile function (QF) vector serves as a "signature" of the image's heterogeneity, which is then processed by a final softmax layer for classification. This approach allows the model to learn complex decision rules, such as identifying a high-grade tumor based on a specific proportion of high-grade patches.

MI Augmentation

To improve robustness and study the effect of spatial context, the authors introduced MI augmentation. During training, the model randomly crops regions of varying sizes from the original image. This forces the network to learn how to aggregate information from different "sub-bags" of tissue, effectively simulating different levels of context.

Overview of the proposed MIL training pipeline using FCN and aggregation.

Experimental Setup

The model was evaluated using a dataset from the Carolina Breast Cancer Study (Phase 3), comprising 5,970 tissue microarray (TMA) cores from 1,713 patients. Five distinct classification tasks were performed using standard Hematoxylin and Eosin (H&E) stained images:

  1. Histologic subtype (ductal vs. lobular)
  2. Estrogen receptor (ER) status
  3. Tumor grade
  4. Risk of recurrence (ROR) score
  5. Genetic subtype (PAM50)

Results

The experimental results highlight two primary advantages of the proposed method: the superiority of quantile aggregation and the importance of spatial context.

Performance Gains from Quantile Aggregation

Quantile aggregation consistently outperformed standard max-pooling. For instance, in the histologic subtype classification task, accuracy improved from 89.8% with max-pooling to 95.2% with quantile aggregation.

The Impact of Spatial Context

A key finding was the dramatic improvement in accuracy as the size of the training crops increased. As shown in Figure 4, moving from single-patch training to larger MI bags significantly reduced the impact of weak label noise. For ER status, accuracy rose from 68.6% to 85.6% as the model was provided with more context during training.

Classification accuracy as a function of cropped image size during training.

Discussion and Interpretability

Beyond classification performance, the MIL framework provides inherent interpretability. Because the model generates predictions at the instance level before aggregation, it is possible to generate heatmaps that visualize the spatial distribution of predicted classes. These visualizations can help pathologists understand the model's reasoning, such as identifying specific regions that contribute to a "ductal" or "lobular" diagnosis. Furthermore, the model's ability to predict molecular features like genetic subtype (54.4% accuracy) and ER status (84.1% accuracy) from standard H&E stains suggests that deep learning can identify subtle morphological signatures of molecular states that are difficult for humans to quantify.

Conclusion

This work demonstrates that modeling tissue heterogeneity through quantile aggregation and MI augmentation significantly improves the performance of CNNs on weakly labeled histopathology images. By moving beyond simple pooling strategies, the proposed framework captures the rich information present in large-scale medical images, paving the way for more accurate and interpretable computer-aided diagnosis in digital pathology.

Summary updated

Abstract

Multiple instance (MI) learning with a convolutional neural network enables end-to-end training in the presence of weak image-level labels. We propose a new method for aggregating predictions from smaller regions of the image into an image-level classification by using the quantile function. The quantile function provides a more complete description of the heterogeneity within each image, improving image-level classification. We also adapt image augmentation to the MI framework by randomly selecting cropped regions on which to apply MI aggregation during each epoch of training. This provides a mechanism to study the importance of MI learning. We validate our method on five different classification tasks for breast tumor histology and provide a visualization method for interpreting local image classifications that could lead to future insights into tumor heterogeneity.
BibTeX
@inproceedings{couture2018multiple,
  author = {Heather D. Couture and J. S. Marron and Charles M. Perou and Melissa A. Troester and Marc Niethammer},
  title = {{Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2018},
  url = {https://openreview.net/forum?id=BIXZsmNRkg5},
}
Figure