Abstract
Spatial pooling (SP) and cross-channel pooling (CCP) operators have been applied to aggregate spatial features and pixel-wise features from feature maps in deep neural networks (DNNs), respectively. Their main goal is to reduce computation and memory overhead without visibly weakening the performance of DNNs. However, SP often faces the problem of losing the subtle feature representations, while CCP has a high possibility of ignoring salient feature representations, which may lead to both miscalibration of confidence issues and suboptimal medical classification results. To address these problems, we propose a novel dual-view framework, the first to systematically investigate the relative roles of SP and CCP by analyzing the difference between spatial features and pixel-wise features. Based on this framework, we propose a new pooling method, termed dual-view pyramid pooling (DVPP), to aggregate multi-scale dual-view features. DVPP aims to boost both medical image classification and confidence calibration performance by fully leveraging the merits of SP and CCP operators from a dual-axis perspective. Additionally, we discuss how to fulfill DVPP with five parameter-free implementations. Extensive experiments on six 2D/3D medical image classification tasks show that our DVPP surpasses state-of-the-art pooling methods in terms of medical image classification results and confidence calibration across different DNNs.
Introduction
In the high-stakes world of medical imaging, deep learning models must be both highly accurate and well-calibrated to ensure clinical safety. However, standard architectures often suffer from "tunnel vision" due to how they aggregate data, frequently leading to overconfident and incorrect diagnoses. This paper introduces Dual-View Pyramid Pooling (DVPP), a novel architectural module that improves both diagnostic performance and model reliability by capturing a more comprehensive set of features.
The Problem: Salient vs. Subtle Features
Deep Neural Networks (DNNs) typically rely on Spatial Pooling (SP), such as Global Average Pooling (GAP), to condense feature maps. While efficient, SP is inherently "lossy," often preserving large-scale "salient" features while washing out the "subtle" details—such as tiny lesions—that are critical for medical diagnosis.
Conversely, Cross-Channel Pooling (CCP) aggregates features across channels at each pixel. While effective at capturing subtle textures, it often misses the broader global context. This imbalance leads to suboptimal accuracy and poor confidence calibration, where models output high confidence scores even for incorrect predictions.
Methodology: Dual-View Pyramid Pooling (DVPP)
The core innovation of DVPP is the systematic integration of SP and CCP into a multi-scale "pyramid" structure. By treating the feature map as a 3D block and extracting information along both spatial and channel axes, DVPP ensures a more holistic representation of the input data.
Key Components:
- Dual-View Concept: Instead of choosing between spatial or channel-wise pooling, DVPP performs both. This dual-axis perspective captures the what and where of structures alongside the intensity and texture of pixels.
- Pyramid Scaling: To account for varying pathology sizes, DVPP uses a pyramid structure (e.g., $1\times1$, $2\times2$, and $4\times4$ scales) to capture features ranging from global organs to microscopic patterns.
- Parameter-Free Implementations: The authors propose five variants (e.g., serial, parallel, and hybrid "Twins" versions) that require no additional learnable weights, maintaining model efficiency and generalization.
Experimental Results
The researchers validated DVPP across six diverse 2D and 3D medical benchmarks, including skin lesions (ISIC2018), brain tumors (BTM), diabetic retinopathy (APTOS2019), and chest X-rays (NIH-CXR-LT).
- Improved Accuracy: On the ISIC2018 task, the SC-DVPP-C-Ser variant improved balanced accuracy by 1.74% and F1-score by 2.18% over standard methods.
- Enhanced Calibration: On the NIH-CXR-LT dataset, DVPP reduced the Expected Calibration Error (ECE) by 13.81%, significantly mitigating model overconfidence.
- Broad Compatibility: The module demonstrated consistent improvements across various backbones, including ResNet, VGG, and Vision Transformers (Swin-T).
Conclusion
DVPP demonstrates that spatial and cross-channel pooling are fundamentally complementary. By providing classifiers with a richer, dual-view feature set, models become more "aware" of the data, leading to both better diagnostic performance and more reliable confidence scores. Future work will explore adapting DVPP for image segmentation and object detection tasks.
Summary updated