MFTrans: A Multi-Resolution Fusion Transformer for Robust Tumor Segmentation in Whole Slide Images

Sungkyu Yang, Woohyun Park, Kwangil Yim, Mansu KimWACV 2025

TL;DR

This paper introduces MFTrans, a novel multi-resolution fusion transformer designed for robust tumor segmentation in whole slide images (WSIs). By mimicking how expert pathologists examine tissue at different magnifications, MFTrans effectively combines local cellular details with global contextual information, achieving state-of-the-art performance on challenging WSI datasets, especially in imbalanced scenarios.

What Problem Does This Paper Address?

Histopathology, the examination of tissue samples, is crucial for cancer diagnosis. However, manual analysis of whole slide images (WSIs) is time-consuming and prone to variability. Existing automated methods often focus on a single magnification level, limiting their ability to capture the full spectrum of information needed for accurate segmentation. Pathologists, on the other hand, skillfully switch between low magnifications for overall structure and high magnifications for detailed cellular morphology. This paper aims to bridge this gap by developing an AI model that can effectively leverage multi-resolution information from WSIs, similar to this expert diagnostic process.

Methods

MFTrans employs a dual-branch architecture inspired by the pathologist's approach:

  • Global Branch: This branch processes low-magnification WSIs using a novel Global Token-based Multi-Head Self-Attention (GT-MSA) Transformer Block. This block efficiently captures global contextual relationships by introducing "global tokens" that aggregate information across windows, reducing computational complexity compared to existing methods.
  • Local Branch: This branch focuses on high-magnification WSIs, utilizing a ConvNeXt block backbone to extract fine-grained local features, such as individual cell morphology.
  • Fusion Branch: A token fusion block with a cross-attention mechanism is used to integrate features from both branches. The global token acts as a query, and local features serve as keys and values, allowing the model to assess globally important local features. This is followed by an image decoder that reconstructs the segmentation mask by combining information from different stages and resolutions.

The model also incorporates a classifier block in the final stage as an auxiliary task to improve discrimination, though it's not used during inference.

Datasets

The paper evaluates MFTrans on three real-world WSI datasets:

  • Camelyon16: A benchmark dataset for detecting metastases in lymph nodes, commonly used for evaluating segmentation algorithms.
  • Catholic Uijeongbu St. Mary's hospital dataset: A dataset collected from a local hospital, used to assess the model's generalizability to data from different sources.
  • PAIP2019: A dataset focused on liver cancer segmentation, used to test the model's robustness across different organs and tissue morphologies.

Key Insights and Ideas

  • Mimicking Pathologists: The core idea of using multi-resolution information, mirroring how expert pathologists work, is a key insight. This approach acknowledges that different levels of detail are crucial for accurate diagnosis.
  • Efficient Global Context: The GT-MSA block is a novel contribution that allows transformers to capture global context more efficiently by using dedicated global tokens, reducing computational burden.
  • Robustness in Imbalance: MFTrans demonstrates superior performance in imbalanced datasets (where normal tissue far outweighs tumor tissue), a common scenario in real-world medical imaging. This highlights its practical applicability.
  • Generalization Across Organs: The model's strong performance on the PAIP dataset (liver cancer) after being trained on lymph node data suggests significant cross-organ generalizability, a critical factor for widespread clinical adoption.

Key Results

MFTrans consistently achieved state-of-the-art or near state-of-the-art performance across all evaluated datasets and scenarios:

  • Camelyon16 (Balanced): Achieved a dice score of 0.918, second only to TransFuse (0.920), but with better Jaccard index (0.848 vs 0.851) and accuracy (0.925).
  • Camelyon16 (Imbalanced, 5:1): MFTrans outperformed all other methods with a dice score of 0.818, Jaccard index of 0.692, and accuracy of 0.938.
  • Catholic Uijeongbu St. Mary's hospital dataset: MFTrans achieved the best performance with a dice score of 0.902, Jaccard index of 0.829, and accuracy of 0.995.
  • PAIP dataset: MFTrans again led with a dice score of 0.801, Jaccard index of 0.688, and accuracy of 0.905.

Qualitatively, visualizations show MFTrans producing more accurate and complete segmentation masks compared to other methods, especially in challenging areas.

Limitations and Future Work

  • Computational Complexity: While MFTrans is more efficient than some transformer-based models, it still involves complex computations. The paper notes that using too many global tokens can degrade performance, suggesting a trade-off between complexity and accuracy.
  • Hyperparameter Tuning: The paper mentions extensive hyperparameter tuning for various models, indicating that achieving optimal performance requires careful configuration.
  • Future Work: The authors suggest further exploration of the number of global tokens and the potential for applying MFTrans to other medical imaging tasks. The successful generalization across different organs also opens avenues for exploring its use in a wider range of clinical applications.

Summary updated

BibTeX
@inproceedings{yang2025mftrans,
  author = {Sungkyu Yang and Woohyun Park and Kwangil Yim and Mansu Kim},
  title = {{MFTrans: A Multi-Resolution Fusion Transformer for Robust Tumor Segmentation in Whole Slide Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure