Single-Stage Broad Multi-Instance Multi-Label Learning (BMIML) with Diverse Inter-Correlations and its application to medical image classification

Qi Lai, Jianhang Zhou, Yanfen Gan, Chi-Man Vong, Deshuang HuangIEEE Trans. Emerg. Top. Comput. Intell 2022 · DOI · OpenReview

Abstract

Multi-instance multi-label (MIML) learning is a powerful paradigm for objects described by multiple instances (e.g., image patches) and simultaneously associated with multiple labels. Existing MIML methods suffer from relatively low accuracy and training efficiency due to several issues: neglected inter-label correlations, the inability to learn inter-instance correlations directly due to missing instance labels, and the requirement of multiple stages to learn diverse correlations. To resolve these issues, we propose Broad Multi-Instance Multi-Label Learning (BMIML), a single-stage framework. BMIML incorporates three innovative modules: Auto-Weighted Label Enhancement Learning (AWLEL) to capture inter-label correlations; Scalable Multi-Instance Probabilistic Regression (SMIPR) to effectively estimate inter-instance correlations using only object labels; and Interactive Decision Optimization (IDO) to combine and optimize results into a single-stage framework. Experiments demonstrate that BMIML is highly competitive in accuracy and significantly faster than existing methods, even for large medical image datasets exceeding 90,000 images.

Introduction: The Complexity of Medical Image Diagnosis

Medical image diagnosis is rarely a simple "one image, one label" task. A single pathology slide or X-ray often contains multiple regions of interest (instances) and can be associated with several different clinical findings (multi-label). This is known as Multi-Instance Multi-Label (MIML) learning.

Existing MIML methods face significant hurdles. They often ignore how different labels relate to each other (e.g., certain diseases often co-occur) and struggle with missing local labels for individual image patches. Furthermore, most current frameworks are multi-stage and computationally expensive, making them slow to train on the large-scale datasets required for modern clinical AI. As shown in the comparison below, BMIML is uniquely designed to capture a comprehensive set of inter-correlations that other methods neglect.

Comparison of diverse inter-correlations captured by various MIML approaches.

The BMIML Framework

BMIML is built upon the Broad Learning System (BLS)—a flat network architecture known for its extreme efficiency compared to deep "tall" networks. The framework integrates global and local views into a single-stage optimization process.

The single-stage BMIML framework architecture, showing the integration of AWLEL, SMIPR, and IDO modules.

Core Modules

  • Auto-Weighted Label Enhancement Learning (AWLEL): This module captures inter-label correlations using a retargeting layer to adaptively adjust label weights, helping the model handle imbalanced data and focus on hard-to-distinguish classes.
  • Scalable Multi-Instance Probabilistic Regression (SMIPR): This module estimates inter-instance correlations. It uses a clustering approach to group similar image bags, allowing the model to estimate the probability that a specific patch belongs to a certain class using only the overall image label.
  • Interactive Decision Optimization (IDO): This module fuses the global view (AWLEL) and the local view (SMIPR) into a single-stage optimization process, preventing information loss typical of disjoint multi-stage steps.

Experimental Results and Scalability

The model was evaluated on five diverse medical imaging datasets, including the NIH Chest X-ray dataset (over 112,000 images) and the ODR retinal fundus dataset.

  • Accuracy: BMIML outperformed seven state-of-the-art methods (including DeepMIML and MIMLSVM) across almost all metrics. On the NIH dataset, it achieved an Average Precision of 0.661, significantly higher than competitors.
  • Efficiency: BMIML was found to be 16.56% to 99.99% faster than existing algorithms. For example, on the ODR dataset, BMIML trained in roughly 19 hours, while some competitors required over 72 hours or failed to return results entirely.
  • Big Data Suitability: Unlike many competitors that "timed out" on the 112K-image NIH dataset, BMIML maintained stable performance and fast training times, showing a clear trend of improving performance as data size increased.

Conclusion and Future Work

BMIML provides a novel, efficient, and accurate framework for complex medical image classification. By integrating diverse inter-correlations into a single-stage broad learning architecture, it overcomes the limitations of traditional multi-stage MIML methods. Future research will explore more sophisticated instance generation techniques, such as ROI detectors, and test the framework's generalization on other complex multi-label data types like video annotation.

Summary updated

Abstract

described by multiple instances (e.g., image patches) and simultaneously associated with multiple labels. Existing MIML methods are useful in many applications but most of which suffer from relatively low accuracy and training efficiency due to several issues: i) the inter-label correlations(i.e., the probabilistic correlations between the multiple labels corresponding to an object) are neglected; ii) the inter-instance correlations (i.e., the probabilistic correlations of different instances in predicting the object label) cannot be learned directly (or jointly) with other types of correlations due to the missing instance labels; iii) diverse inter-correlations (e.g., inter-label correlations, inter-instance correlations) can only be learned in multiple stages. To resolve these issues, a new single-stage framework called broad multi-instance multi-label learning (BMIML) is proposed. In BMIML, there are three innovative modules: i) an auto-weighted label enhancement learning (AWLEL) based on broad learning system (BLS) is designed, which simultaneously and efficiently captures the inter-label correlations while traditional BLS cannot; ii) A specific MIML neural network called scalable multi-instance probabilistic regression (SMIPR) is constructed to effectively estimate the inter-instance correlations using the object label only, which can provide additional probabilistic information for learning; iii) Finally, an interactive decision optimization (IDO) is designed to combine and optimize the results from AWLEL and SMIPR and form a single-stage framework. Experiments show that BMIML is highly competitive to (or even better than) existing methods in accuracy and much faster than most MIML methods even for large medical image data sets (> 90K images).
BibTeX
@article{lai2022single,
  author = {Qi Lai and Jianhang Zhou and Yanfen Gan and Chi-Man Vong and Deshuang Huang},
  title = {{Single-Stage Broad Multi-Instance Multi-Label Learning (BMIML) with Diverse Inter-Correlations and its application to medical image classification}},
  journal = {IEEE Trans. Emerg. Top. Comput. Intell},
  year = {2022},
  doi = {10.1109/tetci.2023.3287978},
}
Figure