TL;DR
This paper introduces "SortingLoss," a novel self-supervised pre-training method for neural networks that leverages the inherent structural order of tomographic (CT) scan slices. By training models to predict the correct sequence of slices from a volume, SortingLoss enables more efficient fine-tuning on sparse annotations, significantly reducing annotation effort and computational costs.
What Problem Does This Paper Address?
Acquiring and annotating large 3D datasets from techniques like Computed Tomography (CT) is a major bottleneck in medical diagnostics and scientific research. Manual annotation is time-consuming, error-prone, and often requires expert knowledge. Current methods typically involve annotating a few 2D slices and then interpolating the rest, which can introduce inaccuracies. While self-supervised learning (SSL) offers a way to learn from unlabeled data, existing SSL methods are often designed for 2D photographic images and don't fully exploit the unique properties of volumetric tomographic data. This paper aims to bridge this gap by developing an SSL method specifically for CT data that reduces the reliance on extensive manual annotation and interpolation.
Methods
The core of the proposed method, "SortingLoss," is a novel pretext task designed for tomographic data. Instead of relying on general-purpose augmentations or complex proxy tasks, SortingLoss assumes that the structural information within a sequence of slices along a specific axis of a CT volume provides enough clues to determine their correct order. The method trains a neural network to predict this order.
Key aspects of the method include:
- Task-Specific Prior: It leverages a "mild assumption" that the anatomical structure within slices along an axis (e.g., cranio-caudal for a human body) can reveal their correct ordering.
- No Negative Sampling: Unlike many contrastive SSL methods (like SimCLR), SortingLoss does not require sampling negative pairs, simplifying the training process and reducing computational overhead.
- Slice-Based Pre-training: The pre-training is performed on individual slices, allowing for efficient fine-tuning on sparse sets of annotated slices without the need for interpolation.
- Margin Ranking Loss: A margin ranking loss is used to enforce that correctly ordered slice pairs have predicted orders that are sufficiently distinct from incorrect ones.
- Mean Displacement Metric: An interpretable metric, "Mean Displacement," is proposed to track the progress of the pre-training, measuring how much slices, on average, would need to be shifted to achieve perfect ordering.
Datasets
The authors evaluated their method on two distinct datasets:
- MosMedData: A dataset containing 1110 volumes of human chest CT scans, with 50 volumes specifically annotated for lung tissues affected by COVID-19.
- Medaka Visual System Dataset: A large-scale dataset of high-resolution synchrotron-based full-body CT scans of Medaka fish (Oryzias latipes), featuring 21 segmentation annotations for structures within the visual system. This dataset was collected as part of an ongoing project and is planned to be released.
Key Insights and Ideas
- Exploiting Volumetric Structure: The paper demonstrates that the inherent 3D structure of tomographic data, specifically the sequential ordering of slices, can be a powerful signal for self-supervised learning, even without explicit 3D supervision.
- Efficiency Gains: By eliminating the need for negative sampling and focusing on a task-specific prior, SortingLoss achieves significant improvements in training speed and memory usage compared to general-purpose SSL methods.
- Reduced Annotation Burden: The ability to fine-tune effectively on sparse slice annotations directly addresses the critical problem of expensive and time-consuming manual annotation for 3D medical data.
- Robustness to Augmentations: The method shows robustness and even improved performance when subjected to tougher data augmentations, suggesting its learned features are more resilient.
- Unsupervised Localization: The learned features can be used to localize samples within volumes in an unsupervised manner, which is a valuable capability for data exploration.
Key Results
- Performance Parity: SortingLoss performs on par with the popular SimCLR baseline on the MosMedData dataset.
- Speed and Memory Efficiency: SortingLoss is 2x faster and requires 1.5x less memory than SimCLR. On a specific GPU, it uses 40% less memory and trains twice as fast.
- Outperformance on Sparse Data: On the Medaka dataset, SortingLoss consistently outperforms SimCLR when fine-tuning on very limited annotated slices (1%, 10%).
- Improved Fine-tuning: Even when fine-tuning on full data, models pre-trained with SortingLoss showed a 2.3% improvement in IoU compared to training from scratch.
- Qualitative Improvements: Visual comparisons (Figure 1) suggest that pre-training with SortingLoss leads to more accurate segmentation of smaller structures.
- Expert Evaluation: While not statistically significant, expert evaluations indicated a preference for segmentations from models pre-trained with SortingLoss over training from scratch.
Limitations and Future Work
- Assumption on Data Structure: The primary limitation is the assumption that the sample's structure provides a strong clue for ordering along at least one axis. This might not hold for all types of tomographic data.
- Future Work:
- Investigating the benefits of other ranking losses that can utilize list-wise ordering information.
- Exploring the application of SortingLoss to datasets from other tomographic imaging methods like MRI, PET, or neutron tomography.
- Further research into the unsupervised localization capabilities.
- The authors plan to release the Medaka visual system dataset after the project concludes.
Summary updated