Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localization

Mingze Yuan, Yingda Xia, Hexin Dong, Zifan Chen, Jiawen Yao, Mingyan Qiu, Ke Yan, Xiaoli Yin, Yu Shi, Xin Chen, Zaiyi Liu, Bin Dong, Jingren Zhou, Le Lu, Ling Zhang 0002, Li Zhang 0047

Introduction

In clinical practice, medical AI models frequently encounter a "long-tail" distribution of diseases. While a model might be proficient at segmenting common conditions like Pancreatic Ductal Adenocarcinoma (PDAC), it often struggles with rare or unseen "out-of-distribution" (OOD) tumors. Standard segmentation frameworks, such as nnUNet, typically operate under a "closed-set" assumption, forcing every pixel into a predefined category. This often leads to misclassification or missed detections of rare pathologies, a challenge known as the "near-OOD" problem.

Problem Statement: The Near-OOD Challenge

The primary difficulty in medical OOD detection is that outliers (rare tumors) often share significant visual similarities with inliers (common tumors) compared to the background. Trustworthy medical AI must accurately segment known diseases while simultaneously localizing suspicious regions that do not match the training data.

Overview of the framework: In-distribution vs. Out-of-distribution (OOD) tasks in medical image segmentation.

Methodology

The authors propose a novel framework based on Mask Transformers, which treat segmentation as a cluster assignment problem using learnable "object queries."

1. MaxQuery for OOD Localization

In this approach, object queries function as prototypical cluster centers. During inference, the model calculates the affinity between image pixels and these queries. The MaxQuery method identifies OOD regions by calculating the maximum similarity score across all inlier queries; a low maximum score indicates the pixel does not belong to any known cluster. The negative of this score serves as an anomaly map for OOD localization.

2. Query-Distribution (QD) Loss

To prevent queries from redundantly focusing on easy-to-segment background regions, the Query-Distribution (QD) loss explicitly partitions queries. It forces a specific subset of queries to focus on the background, the organ, and the tumors, respectively. This ensures high-resolution representation of foreground objects, making it easier to distinguish subtle OOD textures.

3. Architecture

The system utilizes an nnUNet CNN backbone for feature extraction, followed by a transformer decoder that iteratively refines the object queries.

Datasets

The framework was validated on two large-scale, pathologically confirmed 3D CT datasets:

  • Pancreatic Multi-type Tumors (661 patients): Includes 7 common inlier conditions and 3 rare OOD conditions.
  • Liver Multi-type Tumors (427 patients): Includes 5 common inlier conditions and 3 rare OOD conditions.

Key Results and Insights

The proposed method demonstrates significant improvements over baseline OOD detection techniques (e.g., MC Dropout, MaxLogit) and standard segmentation models:

  • OOD Localization: On the pancreatic dataset, the model achieved a 7.39% improvement in AUROC and a 13.79% reduction in the False Positive Rate (FPR95).
  • Inlier Segmentation: The model outperformed the industry-standard nnUNet by an average of 5.27% in Dice Similarity Coefficient (DSC) for known tumor types.
  • Feature-Level Reasoning: Using "pre-softmax" raw affinities proved more effective for OOD detection than post-softmax probabilities, which tend to be overconfident.

Conclusion

By leveraging the query mechanism of Mask Transformers and introducing the Query-Distribution loss, this research provides a robust solution for both high-accuracy segmentation and reliable OOD localization. This dual capability is a critical step toward deploying trustworthy AI in real-world clinical environments where the unexpected is to be expected.

Summary updated

Abstract

Real-world medical image segmentation has tremendous long-tailed complexity of objects, among which tail conditions correlate with relatively rare diseases and are clinically significant. A trustworthy medical AI algorithm should demonstrate its effectiveness on tail conditions to avoid clinically dangerous damage in these out-of-distribution (OOD) cases. In this paper, we adopt the concept of object queries in Mask Transformers to formulate semantic segmentation as a soft cluster assignment. The queries fit the feature-level cluster centers of inliers during training. Therefore, when performing inference on a medical image in real-world scenarios, the similarity between pixels and the queries detects and localizes OOD regions. We term this OOD localization as MaxQuery. Furthermore, the foregrounds of real-world medical images, whether OOD objects or inliers, are lesions. The difference between them is less than that between the foreground and background, possibly misleading the object queries to focus redundantly on the background. Thus, we propose a query-distribution (QD) loss to enforce clear boundaries between segmentation targets and other regions at the query level, improving the inlier segmentation and OOD indication. Our proposed framework is tested on two real-world segmentation tasks, i.e., segmentation of pancreatic and liver tumors, outperforming previous state-of-the-art algorithms by an average of 7.39% on AUROC, 14.69% on AUPR, and 13.79% on FPR95 for OOD localization. On the other hand, our framework improves the performance of inlier segmentation by an average of 5.27% DSC when compared with the leading baseline nnUNet.
BibTeX
@inproceedings{yuan2023devil,
  author = {Mingze Yuan and Yingda Xia and Hexin Dong and Zifan Chen and Jiawen Yao and Mingyan Qiu and Ke Yan and Xiaoli Yin and Yu Shi and Xin Chen and Zaiyi Liu and Bin Dong and Jingren Zhou and Le Lu and Ling Zhang and Li Zhang},
  title = {{Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure