M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in Fundus Images

Qing Liu, Haotian Liu, Yixiong LiangarXiv 2021 · arXiv

Abstract

In medical imaging, particularly diabetic retinopathy screening, detecting tiny lesions in high-resolution fundus images is critical. Standard Convolutional Neural Networks (CNNs) often fail because their downsampling layers—typically Many-to-One Reassembly of Features (M2ORF)—cause small, clustered signals to vanish. We present Many-to-Many Reassembly of Features (M2MRF), a novel architectural component that maps large input regions to multiple output features simultaneously. This approach preserves subtle activations and captures long-range spatial dependencies, significantly improving the segmentation of microaneurysms and other minute pathologies.

Introduction: The Vanishing Signal Problem

The primary challenge in fundus image analysis is extreme scale variation. Lesions such as microaneurysms (MAs) can occupy less than 0.001% of the total pixels in a multi-megapixel image.

Fundus image showing the scale of tiny lesions and their corresponding segmentations.

Traditional CNN architectures rely on downsampling operators like pooling or strided convolutions. These operators follow a Many-to-One paradigm, squashing local neighborhoods into single feature points. When multiple tiny lesions are clustered together, this many-to-one mapping often averages them into the background, leading to a permanent loss of information early in the encoding process. Once these signals "vanish," even sophisticated decoders cannot recover them.

Methodology: Many-to-Many Reassembly

To address the downsampling bottleneck, the authors propose M2MRF. Unlike standard operators that treat each target location independently, M2MRF processes a large predefined region to generate multiple target features at once.

Comparison of Many-to-One (M2ORF) and Many-to-Many (M2MRF) feature reassembly.

Key Architectural Innovations

  1. Many-to-Many Projection: Instead of a standard convolution, M2MRF uses a linear projection mapping $N$ source features to $M$ target features. This allows clustered lesions to reinforce each other's signals through mutual activation enhancement.
  2. Matrix Decomposition: To prevent the parameter explosion associated with large-region projections, the weight matrix is decomposed into two smaller, successive linear projections. This bottleneck structure enables a large receptive field with high efficiency.
  3. Integration: M2MRF was integrated into the HRNetV2 backbone, which is designed for high-resolution representation maintenance. The authors explored both a "Cascade" version (gradual downsampling) and a "One-step" version (direct jump to target resolution).

Experimental Results

The model was evaluated on two major benchmarks: the DDR dataset and the high-resolution IDRiD dataset.

  • Quantitative Performance: On the DDR dataset, the one-step M2MRF-A improved the mean Area Under Precision-Recall curve (mAUPR) from a 45.21% baseline to 49.94%.
  • Comparison with Transformers: M2MRF outperformed modern vision transformers like Swin-Transformer and Twins-SVT on the IDRiD benchmark. It was particularly effective at detecting Microaneurysms (MAs), the smallest and most difficult lesions to segment.
  • Efficiency: Despite the increased receptive field, the matrix decomposition strategy allowed M2MRF to remain more parameter-efficient than existing learning-based operators like CARAFE++.

Conclusion

M2MRF shifts the focus of medical image segmentation from decoder-side upsampling to preserving information during the initial downsampling stages. By adopting a Many-to-Many philosophy, the model prevents the "original sin" of signal washout. While currently optimized for fundus images, this approach holds significant potential for other domains characterized by tiny, clustered objects, such as satellite imagery and microscopy.

Summary updated

Abstract

Feature reassembly is an essential component in modern CNN-based segmentation approaches, which includes feature downsampling and upsampling operators. Existing operators reassemble multiple features from a small predefined region into one for each target location independently. This may result in loss of spatial information, which could vanish activations caused by tiny lesions particularly when they cluster together. In this paper, we propose a many-to-many reassembly of features (M2MRF). It reassembles features in a dimension-reduced feature space and simultaneously aggregates multiple features inside a large predefined region into multiple target features. In this way, long range spatial dependencies are captured to maintain activations on tiny lesions. Experimental results on two lesion segmentation benchmarks, i.e. DDR and IDRiD, show that (1) our M2MRF outperforms existing feature reassembly operators; (2) equipped with our M2MRF, the HRNetv2 is able to achieve significant better performance to CNN-based segmentation methods and competitive even better performance to two recent transformer-based segmentation methods. Our code is made publicly available at https://github.com/CVIU-CSU/M2MRF-Lesion-Segmentation.
BibTeX
@misc{liu2021m,
  author = {Qing Liu and Haotian Liu and Yixiong Liang},
  title = {{M2MRF: Many-to-Many Reassembly of Features for Tiny Lesion Segmentation in Fundus Images}},
  year = {2021},
  eprint = {2111.00193},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.00193},
}
Figure