Multi-Instance Multi-Scale CNN for Medical Image Classification

Shaohua Li, Yong Liu, Xiuchao Sui, Cheng Chen, Gabriel Tjio, Daniel Shu Wei Ting, Rick Siow Mong Goh

Abstract

Classification of medical imagery presents unique challenges, including extreme data scarcity, the "needle in a haystack" problem where small regions of interest (ROIs) are buried in high-resolution scans, and significant scale variance of pathological features. We present the Multi-Instance Multi-Scale (MIMS) CNN, an architecture designed to address these hurdles through a parameter-efficient multi-scale convolutional layer (MSConv) and a learnable top-k pooling mechanism. Evaluated on both 2D and 3D modalities, MIMS-CNN demonstrates state-of-the-art performance, achieving near-perfect classification accuracy while providing interpretable diagnostic visualizations.

Introduction

Deep learning has revolutionized computer vision, yet its application to medical imaging remains non-trivial. Unlike natural image datasets like ImageNet, medical datasets are often small, making models prone to overfitting. Furthermore, diagnostic markers may occupy only a tiny fraction of a high-resolution 3D volume or 2D scan—a challenge exacerbated by "weak labels" where only image-level or volume-level annotations are provided without precise ROI localization. Finally, pathological features like tumors or cysts vary significantly in size, necessitating scale-invariant feature extraction.

Standard approaches, such as image pyramids, are computationally taxing and often fail to maintain feature consistency across scales. MIMS-CNN addresses these issues by processing features at multiple scales within a Multiple Instance Learning (MIL) framework, ensuring robust detection of variable-sized lesions from limited data.

Methodology

The MSConv Layer

To handle scale variance efficiently, MIMS-CNN introduces the Multi-Scale Convolution (MSConv) layer. Rather than resizing the original high-resolution input—which is memory-intensive—the model extracts "primary feature maps" using a pre-trained backbone (e.g., ResNet-101). These compact maps are then resized into multiple scales and processed using a shared set of kernels.

This kernel-sharing strategy is critical: by using a single "template" for a lesion across all scales, the model minimizes parameter count and prevents overfitting. It forces the network to learn scale-invariant representations, bridging the gap where traditional backbones often show poor feature correlation across different input resolutions.

Learnable Top-k Pooling

In the MIL framework, an image is treated as a "bag" of patches or slices. Aggregating these instances into a final classification typically relies on max or average pooling. However, max pooling ignores secondary signals, while average pooling dilutes the signal with healthy tissue.

MIMS-CNN employs a learnable top-k pooling mechanism. It identifies the $k$ highest activations across all scales and instances, calculating a weighted average using learnable parameters. This allows the model to simultaneously "attend" to multiple suspicious regions, improving robustness in cases where pathology is distributed across several slices or patches.

Experimental Results

The MIMS-CNN was rigorously evaluated across four diverse datasets:

  • Cirrus & Spectralis: 3D Optical Coherence Tomography (OCT) for Diabetic Macular Edema (DME) detection.
  • MMD: 2D fundus images for Myopic Macular Degeneration.
  • CRC-MSI: 2D histology images of colorectal cancer.

As shown in the table below, MIMS-CNN consistently outperformed baseline methods and various ablation studies. Notably, it achieved a perfect 1.000 AUROC on the Spectralis dataset and 0.986 on Cirrus.

Performance comparison of MIMS-CNN against various baseline methods across four medical imaging datasets.

Discussion and Insights

A key finding of this work is the importance of magnitude normalization. Because larger scales naturally produce more high-activation pixels, they can dominate the final feature vector. MIMS-CNN utilizes learnable scalar multipliers and batch normalization to balance contributions across scales.

Furthermore, the authors' analysis of feature decorrelation explains why resizing feature maps after the backbone is superior to traditional image pyramids. By forcing the model to find consistent patterns through shared kernels, MIMS-CNN overcomes the inherent scale-inconsistency of standard CNN features.

While highly effective, the model's reliance on the hyperparameter $k$ and pre-trained natural-image backbones suggests avenues for future work. Adapting the framework for other 3D modalities like MRI or CT, where ROIs are even more sparse, remains a promising direction.

Conclusion

MIMS-CNN provides a powerful, parameter-efficient solution for medical image classification under weak supervision. By combining multi-scale feature processing with intelligent aggregation, it bridges the gap between high-resolution data and limited annotations, offering both high accuracy and clinical interpretability.

Summary updated

BibTeX
@inproceedings{li2019multi,
  author = {Shaohua Li and Yong Liu and Xiuchao Sui and Cheng Chen and Gabriel Tjio and Daniel Shu Wei Ting and Rick Siow Mong Goh},
  title = {{Multi-Instance Multi-Scale CNN for Medical Image Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=n81bHtvXVjO},
}
Figure