TL;DR
This paper introduces FourierMIL, a novel multiple instance learning framework that leverages the Discrete Fourier Transform (DFT) to efficiently capture global and local dependencies in gigapixel whole slide images (WSIs) for cancer diagnosis. FourierMIL outperforms existing state-of-the-art methods, demonstrating the power of frequency-domain analysis for complex histopathological image tasks.
What Problem Does This Paper Address?
Analyzing whole slide images (WSIs) in digital pathology presents a significant computational challenge due to their massive size (gigapixels). Standard computer vision models struggle with this scale, and downsampling often leads to the loss of crucial diagnostic information. While multiple instance learning (MIL) has been used to tackle this by treating WSIs as bags of patches, existing methods often overlook essential contextual and hierarchical information. Graph-based methods and transformers have shown promise but face challenges with computational complexity and the unstructured nature of WSI patches. This paper aims to bridge this gap by proposing a method that can effectively capture both global and local dependencies within WSIs without prohibitive computational costs.
Methods
The core of FourierMIL is its novel All-Pass Frequency Filtering (APFF) module for token mixing. Instead of relying solely on spatial domain operations like CNNs or attention mechanisms in transformers, FourierMIL operates in the frequency domain using the Discrete Fourier Transform (DFT).
Here's a breakdown of the key components:
- Feature Extraction: WSIs are first segmented into patches, and features are extracted using pre-trained models like ResNet-50 or CTranspath. These patch features become "tokens."
- Adaptive Token Padding (ATP): To mitigate spectral leakage and edge effects introduced by the DFT on non-periodic image tokens, an adaptive padding strategy is employed. The padding amount is dynamically determined based on the number of original tokens, making it robust to varying input sizes.
- APFF Token Mixing: This is the central innovation. It leverages the convolution theorem, stating that convolution in the spatial domain is equivalent to element-wise multiplication in the frequency domain. The APFF module applies a learnable filter in the frequency domain to mix token features. Crucially, it uses an "all-pass" filter, allowing all frequencies to pass through, preserving both low-frequency (global context) and high-frequency (fine details) information, unlike traditional low-pass or high-pass filters. This is implemented using an MLP-like structure in the frequency domain.
- MLP-Mixer Architecture: FourierMIL builds upon an MLP-Mixer-like architecture, incorporating APFF blocks for token mixing and standard MLP layers for channel mixing. Skip-connections are used to facilitate training.
- Slide-Level Prediction: A learnable class token is appended to the padded tokens. The final state of this class token after passing through the FourierMIL network serves as the slide-level representation, which is then fed into a classification head (a single linear layer) for prediction.
Datasets
The study evaluated FourierMIL on three publicly accessible datasets:
- CAMELYON16 (CM16): Focuses on detecting breast cancer metastasis in lymph node WSIs. This dataset is challenging due to very small cancer regions.
- The Cancer Genome Atlas (TCGA) Non-Small Cell Lung Cancer (NSCLC): Contains WSIs for lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). Tumor regions are larger than in CM16.
- Clinical Proteomic Tumor Analysis Consortium (CPTAC): Used as an external dataset to assess the generalizability of models trained on TCGA-NSCLC.
Key Insights and Ideas
- Frequency Domain Power: Operating in the frequency domain via DFT for token mixing offers a powerful way to capture global dependencies that are often missed by purely spatial methods.
- All-Pass Filtering is Key: The "all-pass" nature of the APFF filter is critical. It allows the model to utilize both global context (low frequencies) and fine details (high frequencies), which are essential for accurate histopathological analysis, especially in cases with sparse tumor regions.
- Adaptive Token Padding: ATP is a clever solution to a fundamental problem with DFT on image data, effectively mitigating spectral leakage and improving the frequency grid's resolution without altering the original image content.
- Efficiency and Effectiveness: FourierMIL achieves strong performance without relying on computationally expensive attention mechanisms, making it more scalable for large WSIs.
Key Results
FourierMIL consistently outperformed state-of-the-art methods across all three datasets:
- CAMELYON16: Achieved a significant improvement of at least 1.9% in accuracy and 3.0% in AUC over existing models. This is particularly impressive given the dataset's challenge of sparse tumor regions.
- TCGA-NSCLC: Showed an increase of at least 1.6% in accuracy and 0.5% in AUC.
- CPTAC: Outperformed competitors by at least 1.3% in accuracy and 0.5% in AUC.
Qualitatively, the paper includes visualizations (Figure 2) showing that FourierMIL's saliency heatmaps closely align with expert pathologist annotations for nodal metastases detection, demonstrating its interpretability and precision in identifying relevant regions, even in challenging cases with minimal tumor presence.
Limitations and Future Work
- Computational Complexity of DFT: While FourierMIL is more efficient than some transformer-based methods, the DFT itself has a computational complexity of O(n log n). The paper notes that Nyströmformer achieves a more favorable O(n) complexity.
- Low Recall: The model demonstrates high precision but exhibits low recall, indicating room for improvement in detecting all relevant cases.
- Future Work: The authors suggest that FourierMIL could play a critical role in advancing tumor detection and classification, supporting more precise diagnostics. Further exploration into improving recall and potentially combining frequency-domain approaches with other advanced techniques could be promising.
Summary updated