Combining convolutional neural networks and self-attention for fundus diseases identification

Keya Wang, Chuanyun Xu, Gang Li, Yang Zhang, Yu Zheng, Chengjie SunScientific 2023 · DOI

TL;DR

This paper introduces MBSaNet, a novel deep learning model that combines Convolutional Neural Networks (CNNs) with Self-Attention (SA) mechanisms to improve the identification of multiple fundus diseases from retinal images. By effectively fusing local features from CNNs with global context from SA, MBSaNet achieves state-of-the-art performance with fewer parameters, offering a promising tool for early and accurate diagnosis of eye conditions.

What Problem Does This Paper Address?

Early detection of fundus diseases is crucial for preventing vision loss and blindness. Fundus photography is a common and effective screening method, but accurately diagnosing multiple diseases simultaneously from these images remains a significant challenge. While deep learning, particularly CNNs, has shown success, their limited ability to capture global relationships in images can hinder performance, especially with complex, multi-label diagnoses. Existing approaches often involve complex ensemble methods that increase computational cost. This paper aims to bridge this gap by developing a more efficient and effective model that leverages both local and global feature extraction capabilities.

Methods

The core innovation of this work is the MBSaNet architecture, which strategically combines CNNs and SA modules.

  • Multiscale Feature Fusion Stem (MFFS): At the initial stage, a novel MFFS is introduced. This component uses convolutional kernels of varying sizes to extract low-level features at different scales and fuses them. This aims to capture richer initial representations of the fundus image.
  • Hybrid Feature Extraction: The network then proceeds through multiple stages. Early stages utilize MBConv blocks (an efficient inverse residual bottleneck block with SE modules) for robust local feature extraction, similar to established CNN architectures.
  • Self-Attention Integration: Crucially, Self-Attention (SA) modules are incorporated in the later stages of the network. These modules are designed to capture long-range dependencies and complex relationships between different spatial locations within the fundus image, compensating for the local focus of CNNs. The SA module is applied after significant downsampling to manage computational complexity.
  • Problem Transformation: For the multilabel classification task, the authors employ a "problem transformation" approach, converting the multilabel problem into multiple binary classification problems, one for each disease.

Datasets

The primary dataset used for training and evaluation is the ODIR-5K dataset. This dataset is described as containing "real" patient data with color fundus images and diagnostic keywords from various hospitals and medical centers in China. It includes a large number of images with varying resolutions and camera types. The dataset is noted for its imbalance in class distribution. The authors also created a balanced test set from the training data for more robust evaluation.

Key Insights and Ideas

  • Synergy of CNNs and SA: The paper strongly advocates for the combined power of CNNs and SA. The intuition is that CNNs excel at local feature extraction and inductive bias (leading to better generalization on smaller datasets), while SA excels at capturing global context and long-range dependencies. MBSaNet demonstrates that this combination can outperform models relying solely on one approach.
  • Importance of Multiscale Fusion: The introduction of the Multiscale Feature Fusion Stem (MFFS) highlights the benefit of extracting features at different scales early in the network. This suggests that diverse lesion sizes and characteristics in fundus images are better captured by considering multiple receptive fields from the outset.
  • Strategic SA Placement: The decision to place SA modules in the later stages, after significant downsampling, is a practical consideration. This approach balances the computational cost of SA (which can be quadratic with respect to spatial size) with its benefit of capturing global context, preventing it from overwhelming the model's resources.
  • Addressing Multilabel Challenges: The explicit mention of transforming the multilabel problem into binary classifications for each label is a standard but important technique for handling complex diagnostic scenarios where a single image can indicate multiple diseases.

Key Results

  • State-of-the-Art Performance: MBSaNet achieved state-of-the-art performance on the ODIR-5K dataset across various metrics, including AUC, Kappa, and F1-score.
  • Improved AUC: Notably, MBSaNet achieved a significantly higher AUC (0.891 on the off-site test set) compared to the best-performing CNNs (0.870), which is particularly important for imbalanced datasets like ODIR-5K.
  • Parameter Efficiency: The model achieved these superior results with fewer parameters compared to many other models, making it more computationally efficient. For instance, on the off-site test set, MBSaNet achieved a final score of 0.737 with only 9.4M parameters, outperforming models with significantly more parameters.
  • Robustness: The model demonstrated applicability across a wide range of diseases and different fundus image collection conditions, indicating its robustness.
  • Ablation Study Findings: Ablation studies confirmed that horizontally widening the stem structure improved performance over vertical stacking, and that the multiscale feature fusion stem was more effective than single-scale approaches. Introducing global SA in the final stage also significantly improved feature modeling ability.

Limitations and Future Work

  • Dataset Imbalance: The ODIR-5K dataset is inherently imbalanced, which can make it difficult to assess the model's performance on less frequent diseases. The authors acknowledge that the "Other" category, containing diverse diseases, poses a challenge due to large intra-class gaps.
  • Low-Quality Images: While some low-quality images were removed, the authors note that such images are unavoidable in practical situations and could impact real-world performance.
  • Limited Number of Images in Some Categories: The paper mentions that a limited number of images in certain categories might affect model performance, despite data augmentation.
  • Future Work: The authors suggest further research into the impact of hidden dimension sizes in the convolution stage and the number of fully-connected layers, as these factors were found to influence recognition accuracy. Exploring more advanced data augmentation techniques and investigating the impact of different camera types could also be beneficial.

Summary updated

Abstract

Early detection of lesions is of great significance for treating fundus diseases. Fundus photography is an effective and convenient screening technique by which common fundus diseases can be detected. In this study, we use color fundus images to distinguish among multiple fundus diseases. Existing research on fundus disease classification has achieved some success through deep learning techniques, but there is still much room for improvement in model evaluation metrics using only deep convolutional neural network (CNN) architectures with limited global modeling ability; the simultaneous diagnosis of multiple fundus diseases still faces great challenges. Therefore, given that the self-attention (SA) model with a global receptive field may have robust global-level feature modeling ability, we propose a multistage fundus image classification model MBSaNet which combines CNN and SA mechanism. The convolution block extracts the local information of the fundus image, and the SA module further captures the complex relationships between different spatial positions, thereby directly detecting one or more fundus diseases in retinal fundus image. In the initial stage of feature extraction, we propose a multiscale feature fusion stem, which uses convolutional kernels of different scales to extract low-level features of the input image and fuse them to improve recognition accuracy. The training and testing were performed based on the ODIR-5k dataset. The experimental results show that MBSaNet achieves state-of-the-art performance with fewer parameters. The wide range of diseases and different fundus image collection conditions confirmed the applicability of MBSaNet.
BibTeX
@inproceedings{wang2023combining,
  author = {Keya Wang and Chuanyun Xu and Gang Li and Yang Zhang and Yu Zheng and Chengjie Sun},
  title = {{Combining convolutional neural networks and self-attention for fundus diseases identification}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-022-27358-6},
}
Figure