Abstract
Segmentation of distinct bones plays a crucial role in diagnosis, planning, navigation, and the assessment of bone metastasis. It supplies semantic knowledge to visualisation tools for the planning of surgical interventions and the education of health professionals. Fully supervised segmentation of 3D data using Deep Learning methods has been extensively studied for many tasks but is usually restricted to distinguishing only a handful of classes. With 125 distinct bones, our case includes many more labels than typical 3D segmentation tasks. For this reason, the direct adaptation of most established methods is not possible. This paper discusses the intricacies of training a 3D segmentation network in a many-label setting and shows necessary modifications in network architecture, loss function, and data augmentation. As a result, we demonstrate the robustness of our method by automatically segmenting over one hundred distinct bones simultaneously in an end-to-end learnt fashion from a CT-scan.
The Problem: Scaling to 125 Classes
In medical imaging, 3D segmentation is typically a "few-label" task. Most models are trained to identify a handful of structures, such as the liver, a tumor, and the background. However, the human skeleton is composed of over 200 bones. Segmenting them individually is vital for surgical planning, navigation, and tracking diseases like bone metastasis.
The challenge of segmenting 125 distinct bones simultaneously is twofold:
- Memory Constraints: In a standard 3D U-Net, the final classification layer's size is proportional to the number of classes. With 125 labels, the memory required for the output volume ($H \times W \times D \times \text{Classes}$) quickly exceeds the RAM of even high-end GPUs.
- Class Imbalance: Large structures like the pelvis or spine provide massive amounts of data, while tiny bones in the hand are easily "ignored" by the network during training, leading to poor detection rates.
Methodology: Architectural and Algorithmic Adjustments
To address these challenges, researchers from the University of Basel utilized a modified 3D U-Net architecture called No-New-Net, which reduces trainable parameters to allow for larger input patches. To make the "many-label" task feasible, they implemented several key technical adjustments:
- Sparse Representations: To avoid Out-of-Memory (OOM) errors, they used sparse representations for class-wise predictions and ground truths during loss calculation.
- Hybrid Loss Function: They employed an unweighted linear combination of multi-class cross-entropy and Dice Similarity Coefficient (DSC) loss to handle the high imbalance between bone and background voxels.
- Balanced Sampling: Instead of picking random sub-volumes (patches) from the CT scan, they used a sampling strategy where every bone class has an equal probability of being present in a training patch. This ensures the network "sees" small bones as often as large ones.
- Patch-Based Training: They tested various patch sizes (up to $160^3$ pixels) on GPUs with different RAM capacities (8GB to 24GB), balancing the need for global anatomical context with computational limits.
Key Results and Performance
The model, specifically the version using $160^3$ patches and balanced sampling (160_bal,d), showed impressive robustness in segmenting complex skeletal anatomy in roughly one minute.
- Large Bones & Spine: Achieved median DSC scores of 0.82 to 0.90, with a 100% detection ratio.
- Ribs: Achieved a median DSC of 0.58. While lower, this is expected given the high visual similarity between adjacent ribs within a local patch.
- Hands: This was the most difficult region, with a detection ratio of roughly 0.59. Many tiny bones were still missed or mislabeled due to their size and proximity.
Conclusion and Future Directions
The paper highlights that established 3D segmentation methods hit a "scaling wall" when adding more classes; the classification layer itself becomes the primary bottleneck. Balanced sampling proved non-negotiable for many-label tasks, as uniform random sampling fails to provide enough data for smaller structures.
The primary limitation remains the small dataset size (5 scans), which limits generalization across diverse body types. Future work will focus on incorporating shape priors to help the network distinguish between similar-looking adjacent structures and using the current model to generate "initial guesses" for unlabelled data to accelerate the manual annotation process.
Summary updated