Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Bone Shape Reconstruction

Mahesh Shakya, Bishesh KhanalNeurIPS 2023

Introduction

Reconstructing 3D bone structures from 2D X-rays is a "holy grail" for orthopedics. It offers the spatial detail of a CT scan combined with the low radiation, low cost, and portability of an X-ray. However, the field has historically been a "Wild West" of non-reproducible research. Most papers use private datasets, evaluate only one type of bone (e.g., just the femur), and rely solely on geometric metrics like the Dice Similarity Coefficient (DSC).

This paper introduces the first comprehensive benchmarking framework for reconstructing 3D bone shapes from biplanar (orthogonal) X-rays. By standardizing eight deep learning models across six public datasets, the authors address four major gaps in the field: dataset fragmentation, lack of reproducibility, clinical irrelevance of standard metrics, and poor real-world robustness.

Methodology

To solve these issues, the researchers built an open-source platform called xrayto3D-benchmark.

Overview of the xrayto3D-benchmark pipeline

The platform standardizes the end-to-end process and includes:

  • Standardized Pipeline: Tools to convert CT scans into Digitally Reconstructed Radiographs (DRRs)—simulated X-rays used for training and testing.
  • Model Zoo: Reference implementations of eight representative encoder-decoder architectures, including CNN-based models (like UNet and 2DConcat) and Transformer-based models (like SwinUNETR).
  • Clinical Metric Extraction: Automated scripts to calculate clinically vital parameters, such as the Neck-Shaft Angle (NSA) for femurs and Vertebra Canal Length (VCL) for spines.

Datasets

The benchmark unifies six public CT segmentation datasets covering four distinct anatomies (Hip, Vertebra, Femur, and Rib):

  • TotalSegmentator: Used to derive the Rib, Femur, and Pelvic datasets.
  • CTPelvic1k: Includes subsets for "Clinic" (fractures) and "Clinic-metal" (implants).
  • VerSe2019: Utilized for individual vertebrae reconstruction.
  • RSNACervicalFracture: Specifically included for testing model robustness on fractured cervical spines.

Key Insights

  • The Dice Paradox: This is the paper's most significant finding. A model might achieve a high Dice score by getting the overall "blob" of the bone right, yet fail miserably at reconstructing a specific landmark needed for surgery. Conversely, a lower-scoring model might preserve the bone's functional geometry better, highlighting the need for specialized clinical metrics.
  • Attention is Essential: Models that use self-attention mechanisms (like SwinUNETR) or gating (like AttentionUNet) consistently outperformed pure CNNs. They are better at capturing the global spatial relationships necessary to map 2D projections back to 3D space.
  • Ribs are the Final Boss: While femurs and hips are relatively easy to reconstruct due to their distinct shapes, ribs are significantly harder. Their thin, repetitive, and highly curved structure makes them a major challenge for current architectures.
  • The Fragility of "Clean" AI: Models trained on healthy bones see a massive performance drop when faced with fractures or metal implants. This "domain shift" is a primary barrier to using these models in actual trauma centers.

Key Results

Overall, SwinUNETR and AttentionUNet led the pack across most tasks.

Model ranking across different anatomies
  • Top Performers: On the Femur dataset, SwinUNETR achieved a 93.64% Dice score, while AttentionUNet achieved 93.66%.
  • Efficiency Matters: Interestingly, AttentionUNet achieved state-of-the-art results with only 1.5 million parameters, compared to SwinUNETR’s 62.2 million, making it far more viable for deployment on modest hardware.
  • Robustness Issues: When the lateral X-ray view was tilted by just 10 degrees (simulating a technician's error), the Dice score for ribs dropped by as much as 30%, though vertebrae models proved more resilient to these misalignments.

Limitations and Future Work

The primary limitation of current methodologies is the reliance on DRRs (simulated X-rays). While DRRs are standard for training, real X-rays contain noise, soft tissue interference, and varying exposure levels that DRRs do not perfectly mimic.

The authors suggest that future research should move away from optimizing for "pixel-perfect" overlaps and instead focus on clinical parameter-aware loss functions—essentially teaching the AI that some parts of the bone (like a joint surface) are more important to get right than others. They also call for the community to contribute more "paired" datasets (real X-rays and CTs from the same patient) to bridge the gap to clinical reality.

Summary updated

BibTeX
@inproceedings{shakya2023benchmarking,
  author = {Mahesh Shakya and Bishesh Khanal},
  title = {{Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Bone Shape Reconstruction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure