Classification of Medical Images and Illustrations in the Biomedical Literature Using Synergic Deep Learning

Jianpeng Zhang, Yong Xia, Qi Wu, Yutong XiearXiv 2017 · arXiv

Abstract

Automating the classification of medical images in biomedical literature is a significant challenge due to high intra-class variation and substantial inter-class similarity. This paper introduces Synergic Deep Learning (SDL), a novel framework that utilizes a dual-network architecture linked by a "synergic signal." This signal enables the model to learn from pairs of images, effectively distinguishing between confusingly similar modalities. Evaluated on the ImageCLEF2016 benchmark, the SDL model achieved a state-of-the-art accuracy of 86.58%, outperforming deeper standalone models and setting a new record for the challenge.

Introduction

The classification of medical literature images is hindered by two primary visual hurdles. First, intra-class variation occurs when images in the same category (e.g., Computerized Tomography) look vastly different because they scan different anatomical regions. Second, inter-class similarity occurs when images from different categories (e.g., a brain CT vs. a brain MRI) share identical anatomical structures, making modality identification difficult for standard Deep Convolutional Neural Networks (DCNNs).

Examples of medical images (CT and MRI) illustrating the challenges of intra-class variation and inter-class similarity.

Synergic Deep Learning Framework

The core innovation of this work is the SDL model, which moves beyond standard ensemble methods by breaking the independence of networks during the training phase.

Dual-DCNN and Synergic Signal

The architecture consists of two identical DCNNs (e.g., ResNet-50). During training, the model is fed pairs of images. The "Synergic Signal System" acts as an information bridge that checks if the pair belongs to the same category.

The Synergic Deep Learning (SDL) architecture, featuring dual DCNNs and the synergic signal system for pair-based learning.

Corrective Feedback

If the two networks produce a "synergic error"—such as disagreeing on a pair or failing to recognize a match—this error is backpropagated to both networks. This forces the models to mutually learn better representations, specifically targeting the "confusing" samples that typically lead to misclassification. During the inference phase, the synergic signal is discarded, and the final prediction is made by averaging the likelihood probabilities from both DCNN components.

Evaluation and Results

The authors evaluated the SDL model using the ImageCLEF2016 Subfigure Classification Challenge dataset, which contains 30 distinct classes including 12 medical diagnostic modalities and 18 types of illustrations.

  • State-of-the-Art Performance: The SDL model achieved a classification accuracy of 86.58% on the test set, surpassing the previous first-place solution (85.38%) which utilized a deeper ResNet-152 model.
  • Synergic Boost: Ablation studies showed that the synergic signal improved the accuracy of individual ResNet-50 components by more than 2% compared to standard standalone training.
  • Robustness: The model demonstrated significant improvements in F-scores across almost all 30 categories, particularly those with high inter-class similarity.

Discussion and Future Work

SDL provides a scalable, backbone-agnostic framework to boost DCNN performance by focusing on sample relationships. While effective, the simultaneous training of two networks increases computational overhead. Future research could investigate "hard-negative mining" to select the most confusing image pairs and applying the synergic framework to more modern architectures like Vision Transformers.

Summary updated

Abstract

The Classification of medical images and illustrations in the literature aims to label a medical image according to the modality it was produced or label an illustration according to its production attributes. It is an essential and challenging research hotspot in the area of automated literature review, retrieval and mining. The significant intra-class variation and inter-class similarity caused by the diverse imaging modalities and various illustration types brings a great deal of difficulties to the problem. In this paper, we propose a synergic deep learning (SDL) model to address this issue. Specifically, a dual deep convolutional neural network with a synergic signal system is designed to mutually learn image representation. The synergic signal is used to verify whether the input image pair belongs to the same category and to give the corrective feedback if a synergic error exists. Our SDL model can be trained 'end to end'. In the test phase, the class label of an input can be predicted by averaging the likelihood probabilities obtained by two convolutional neural network components. Experimental results on the ImageCLEF2016 Subfigure Classification Challenge suggest that our proposed SDL model achieves the state-of-the art performance in this medical image classification problem and its accuracy is higher than that of the first place solution on the Challenge leader board so far.
BibTeX
@misc{zhang2017classification,
  author = {Jianpeng Zhang and Yong Xia and Qi Wu and Yutong Xie},
  title = {{Classification of Medical Images and Illustrations in the Biomedical Literature Using Synergic Deep Learning}},
  year = {2017},
  eprint = {1706.09092},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.09092},
}
Figure