RemixFormer: A Transformer Model for Precision Skin Tumor Differential Diagnosis via Multi-modal Imaging and Non-imaging Data

Jing Xu, Yuan Gao, Wei Liu, Kai Huang, Shuang Zhao, Le Lu, Xiaosong Wang, Xian-Sheng Hua, Yu Wang, Xiang ChenMICCAI 2022 · OpenReview

Abstract

Skin tumor is one of the most common diseases worldwide and the survival rate could be drastically increased if the cancerous lesions were identified early. Intrinsic visual ambiguities displayed by skin tumors in multi-modal imaging data impose huge amounts of challenges to diagnose them precisely, especially at the early stage. To achieve high diagnosis accuracy or precision, all possibly available clinical data (imaging and/or non-imaging) from multiple sources are used, and even the missing-modality problem needs to be tackled when some modality may become unavailable. To this end, we first devise a new disease-wise pairing of all accessible patient data if they fall into the same disease category as a remix operation of data samples. A novel cross-modality-fusion module is also proposed and integrated with our transformer-based multi-modality deep classification framework that can effectively perform multi-source data fusion (i.e., clinical images, dermoscopic images and accompanied with clinical patient-wise metadata) for skin tumors. Extensive quantitative experiments are conducted. We achieve an absolute 6.5% increase in averaged F1 and 2.8% in accuracy for the classification of five common skin tumors by comparing to the prior leading method on Derm7pt dataset of 1011 cases. More importantly, our method obtains an overall 88.5% classification accuracy using a large-scale in-house dataset of 5601 patients and in ten skin tumor classes (pigmented and non-pigmented). This experiment further validates the robustness and implies the potential clinical usability of our method, in a more realistic and pragmatic clinic setting.
BibTeX
@inproceedings{xu2022remixformer,
  author = {Jing Xu and Yuan Gao and Wei Liu and Kai Huang and Shuang Zhao and Le Lu and Xiaosong Wang and Xian-Sheng Hua and Yu Wang and Xiang Chen},
  title = {{RemixFormer: A Transformer Model for Precision Skin Tumor Differential Diagnosis via Multi-modal Imaging and Non-imaging Data}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  url = {https://openreview.net/forum?id=xzspF74R8N},
}

Loading related content...

Figure