CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans

Jieneng Chen, Yingda Xia, Jiawen Yao, Ke Yan, Jianpeng Zhang, Le Lu, Fakai Wang, Bo Zhou, Mingyan Qiu, Qihang Yu, Mingze Yuan, Wei Fang, Yuxing Tang, Minfeng Xu, Jian Zhou, Yuqian Zhao, Qifeng Wang, Xianghua Ye, Xiaoli Yin, Yu Shi, Xin Chen, Jingren Zhou, Alan Yuille, Zaiyi Liu, Ling Zhang

TL;DR

CancerUniT is a unified transformer-based model that performs detection, segmentation, and diagnosis for eight major cancer types in CT scans within a single framework. By employing a novel hierarchical query architecture that mimics clinical reasoning, it significantly outperforms ensembles of specialized single-organ models while being 4.5x faster and 8x lighter.

What Problem Does This Paper Address?

In clinical practice, radiologists perform "full-body" reads, scanning an entire CT volume for any abnormalities across multiple organs. However, most medical AI systems are "siloed"—designed to detect a specific disease in a single organ (e.g., just lung nodules or just liver lesions).

To match a human radiologist's workflow, clinicians currently have to assemble a complex "committee" of independent AI models. This approach is computationally expensive, difficult to integrate, and ignores the anatomical relationships between organs. Furthermore, existing multi-lesion datasets often lack the high-quality 3D masks and pathology-confirmed labels necessary for precise diagnosis. CancerUniT aims to bridge this gap by providing a single, universal model for multi-organ cancer screening.

We aim at cancer and non-cancer detection, segmentation, and diagnosis in eight major organs via CT scan.

Methods

The core of the paper is the Unified Tumor Transformer (CancerUniT). It builds upon the Mask Transformer architecture but introduces a clinically-inspired Query Hierarchy:

  1. Hierarchical Queries: Instead of treating all classes equally, the model decouples object queries into three groups:
  2. Organ Queries: Identify the anatomical context (e.g., "This is the liver").
  3. Detection Queries: Identify the presence of a general tumor (e.g., "There is a mass here").
  4. Diagnosis Queries: Classify the specific subtype (e.g., "This mass is a Hepatocellular Carcinoma").
  5. Detection-to-Diagnosis Hierarchy: The model uses linear projections to establish parent-child relationships between detection queries and diagnosis queries. This forces the model to learn that specific cancer subtypes are subsets of broader tumor categories.
  6. Dual-Task Decoding: To handle the fact that a voxel can be both a "liver tumor" (detection) and "HCC" (diagnosis), the model decodes these tasks separately. This allows it to enforce class exclusivity within each task (via softmax) while maintaining the relationship between them.
  7. Backbone: The system uses a standard nnUNet as a feature extractor, which is then processed by the Transformer decoder layers to update the queries based on image features.
Figure 1. We aim at cancer and non-cancer detection, segmenta- tion, and diagnosis in eight major organs via CT scan. Seven of our eight targeted cancers rank the top seven in terms of mortality.

Datasets

The authors curated a massive, high-quality dataset to train and validate CancerUniT:

  • Primary Dataset: 10,673 patients in total. It includes seven in-house datasets from five hospitals (pathology-confirmed with 3D masks) and one public kidney tumor dataset (KiTS19).
  • Scope: Covers eight organs (lung, colorectum, liver, stomach, breast, esophagus, pancreas, and kidney). These represent seven of the top mortality-ranking cancers globally.
  • Diversity: Includes both malignant cancers and benign (non-cancer) tumors, which is critical for reducing false positives in real-world screening.
  • Generalizability Testing: The model was also tested on the public Medical Segmentation Decathlon (MSD) datasets for lung, liver, and pancreas to prove it works on unseen data distributions.

Key Insights and Ideas

  • Unified Learning is a Force Multiplier: Training on multiple organs simultaneously actually improves performance on individual organs. The model learns anatomical constraints (e.g., an HCC must be in the liver) and shared visual features across different tumor types.
  • Hierarchy Mimics Expertise: By explicitly structuring queries to follow a "detect then diagnose" path, the model achieves better diagnostic accuracy than "flat" models that try to classify everything at once.
  • Efficiency Matters for Adoption: Moving from an ensemble of eight models to one unified model reduces the memory footprint by 8x. This makes it much more feasible to deploy in standard hospital IT environments.

Key Results

CancerUniT demonstrated superior performance across nearly all metrics compared to state-of-the-art baselines like DeepLesion and ensembles of nnUNet models:

  • Detection Sensitivity: Achieved an average sensitivity of 93.3%, significantly higher than the 8-nnUNet ensemble (86.6%) and DeepLesion (64.0%).
  • Specificity: On normal (cancer-free) patients, it reached 81.7% specificity, a massive jump from the 59.2% achieved by the nnUNet ensemble.
  • Diagnostic Accuracy: Reached 70.9% average accuracy in differentiating subtypes, outperforming the next best model (68.1%).
  • Computational Speed: The model processes scans in 42 seconds, compared to 187 seconds for an ensemble of expert models.
Qualitative results of full spectrum tumors in the test set being segmented and diagnosed by CancerUniT.

Limitations and Future Work

  • Data Imbalance: Some organs, like the breast (478 patients), had significantly less data than others like the lung (2,402 patients), which can lead to biased performance.
  • Multi-Task Trade-offs: While the unified model is better overall, the authors noted a slight performance drop in breast cancer detection (-2%) compared to a dedicated single-organ model, suggesting that multi-task interference is still a challenge.
  • Supervision Granularity: The model currently relies on voxel-wise masks. The authors suggest that incorporating higher-level clinical report data could further refine the fine-grained diagnosis of complex subtypes.

Summary updated

Abstract

Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice, while most medical AI systems are built to focus on single organs with a narrow list of a few diseases. This might severely limit AI's clinical adoption. A certain number of AI models need to be assembled non-trivially to match the diagnostic process of a human reading a CT scan. In this paper, we construct a Unified Tumor Transformer (CancerUniT) model to jointly detect tumor existence & location and diagnose tumor characteristics for eight major cancers in CT scans. CancerUniT is a query-based Mask Transformer model with the output of multi-tumor prediction. We decouple the object queries into organ queries, tumor detection queries and tumor diagnosis queries, and further establish hierarchical relationships among the three groups. This clinically-inspired architecture effectively assists inter- and intra-organ representation learning of tumors and facilitates the resolution of these complex, anatomically related multi-organ cancer image reading tasks. CancerUniT is trained end-to-end using a curated large-scale CT images of 10,042 patients including eight major types of cancers and occurring non-cancer tumors (all are pathology-confirmed with 3D tumor masks annotated by radiologists). On the test set of 631 patients, CancerUniT has demonstrated strong performance under a set of clinically relevant evaluation metrics, substantially outperforming both multi-disease methods and an assembly of eight single-organ expert models in tumor detection, segmentation, and diagnosis. This moves one step closer towards a universal high performance cancer screening tool.
BibTeX
@inproceedings{chen2023cancerunit,
  author = {Jieneng Chen and Yingda Xia and Jiawen Yao and Ke Yan and Jianpeng Zhang and Le Lu and Fakai Wang and Bo Zhou and Mingyan Qiu and Qihang Yu and Mingze Yuan and Wei Fang and Yuxing Tang and Minfeng Xu and Jian Zhou and Yuqian Zhao and Qifeng Wang and Xianghua Ye and Xiaoli Yin and Yu Shi and Xin Chen and Jingren Zhou and Alan Yuille and Zaiyi Liu and Ling Zhang},
  title = {{CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01950},
}
Figure