Abstract
Self-supervised learning (SSL) opens up huge opportunities for medical image analysis that is well known for its lack of annotations. However, aggregating massive (unlabeled) 3D medical images like computerized tomography (CT) remains challenging due to its high imaging cost and privacy restrictions. In this paper, we advocate bringing a wealth of 2D images like chest X-rays as compensation for the lack of 3D data, aiming to build a universal medical self-supervised representation learning framework, called UniMiSS. The following problem is how to break the dimensionality barrier, i.e., making it possible to perform SSL with both 2D and 3D images? To achieve this, we design a pyramid U-like medical Transformer (MiT). It is composed of the switchable patch embedding (SPE) module and Transformers. The SPE module adaptively switches to either 2D or 3D patch embedding, depending on the input dimension. The embedded patches are converted into a sequence regardless of their original dimensions. The Transformers model the long-term dependencies in a sequence-to-sequence manner, thus enabling UniMiSS to learn representations from both 2D and 3D images. With the MiT as the backbone, we perform the UniMiSS in a self-distillation manner. We conduct expensive experiments on six 3D/2D medical image analysis tasks, including segmentation and classification. The results show that the proposed UniMiSS achieves promising performance on various downstream tasks, outperforming the ImageNet pre-training and other advanced SSL counterparts substantially.
Introduction: Breaking the Dimensionality Barrier
Deep learning in medicine faces a chronic "data hunger" problem. While millions of medical images exist, very few are labeled by experts. Self-supervised learning (SSL)—where models learn from unlabeled data by solving "pretext" tasks—is a popular solution. However, medical imaging faces a unique "dimensionality barrier." High-stakes diagnostic data is often 3D (e.g., CT or MRI scans), which is expensive and scarce. Conversely, 2D data (e.g., Chest X-rays) is abundant but structurally different.
Traditionally, researchers have built separate models for 2D and 3D, preventing 3D models from benefiting from the massive wealth of 2D data. Convolutional Neural Networks (CNNs) are particularly rigid; a 2D CNN cannot process a 3D volume without significant architectural changes. UniMiSS addresses this by creating a universal framework that treats 2D images as "fuel" for 3D learning.
The UniMiSS Framework
The core of UniMiSS is a "dimension-free" architecture designed to bridge the gap between 2D and 3D data.
Pyramid U-like Medical Transformer (MiT)
Instead of rigid CNNs, UniMiSS utilizes a Vision Transformer. Transformers are inherently flexible because they treat input data as a sequence of "tokens" (patches), regardless of the input's original dimensionality. This sequence-to-sequence modeling allows the model to learn long-term dependencies across different data formats.
Switchable Patch Embedding (SPE)
The SPE module is the architectural innovation that enables multi-dimensional learning. It adaptively switches between 2D and 3D convolution to convert input images into standardized sequences. Whether the input is a flat X-ray or a volumetric CT, the SPE module ensures the Transformer receives a compatible sequence.
Self-Distillation and Consistency
UniMiSS employs a student-teacher setup (similar to the DINO framework), where a student network learns to match the output of a teacher network seeing different augmented views. To further boost 3D learning, the authors introduced a volume-slice consistency constraint, forcing the model to recognize that a 3D volume and its constituent 2D slices represent the same anatomy.
Experimental Results and Insights
The model was pre-trained on a massive mixed-dimension pool of 5,022 unlabeled CT scans and 108,948 unlabeled Chest X-rays. Evaluation across six downstream tasks (3D multi-organ segmentation, COVID-19 screening, 2D lung segmentation, etc.) revealed several key findings:
- Label Efficiency: On the BCV 3D segmentation task, UniMiSS trained with only 40% of the labels outperformed a randomly initialized model using 100% of the labels.
- Superior 3D Performance: In 3D segmentation, UniMiSS achieved an 84.99 Dice score, significantly higher than the 82.61 achieved by DINO (using only 3D data).
- Cross-Modality Transfer: Despite being pre-trained on CT and X-ray, the model performed exceptionally well on MRI and Dermoscopy images, proving the learned features are highly transferable across medical modalities.
Conclusion
UniMiSS demonstrates that 2D X-rays can effectively compensate for the scarcity of 3D data in medical AI. By leveraging Transformers and the Switchable Patch Embedding module, the framework breaks the dimensionality barrier, creating a universal representation of human anatomy. Future work may extend this to incorporate 1D clinical text or genetic data, further unifying medical data analysis.
Summary updated