Anatomy-Aware Inference of the 3D Standing Spine Posture from 2D Radiographs

Amirhossein Bayat, Danielle F. Pace, Anjany Sekuboyina, Christian Payer, Darko Štern, Martin Urschler, Jan S. Kirschke, Bjoern MenzeTomography 2022 · DOI

TL;DR

This paper introduces TransVert+, a deep learning model that reconstructs a patient's 3D spine posture from standard 2D X-ray images. By incorporating anatomical shape priors and a novel registration-based approach, it overcomes the limitations of existing methods that require patients to lie down for 3D scans, enabling more accurate analysis of spine health in a natural, weight-bearing stance.

What Problem Does This Paper Address?

Analyzing the spine's balance and load-bearing capacity is crucial for understanding spinal degeneration, pain, and surgical outcomes. This analysis ideally requires a patient to be in an upright, standing position to reflect physiological loading. However, current 3D imaging techniques like CT and MRI necessitate patients to be in a prone or supine position, which doesn't accurately represent the spine's natural curvature and load distribution under weight. While 2D radiographs can capture the spine in a standing position, they inherently lose 3D spatial information, making direct 3D reconstruction challenging. Existing methods for 3D reconstruction from 2D images often rely on manual registration, are time-consuming, or lack the ability to capture the complex, patient-specific 3D spinal shape in an upright stance.

Methods

The core of this work is the TransVert+ architecture, an advancement over their previous TransVert model. It tackles the 3D spine reconstruction problem as a registration task, where vertebral shape templates are deformed to match the target 3D shape.

Key methodological components include:

  • Anatomy-Aware Deep Neural Network: TransVert+ fuses information from orthogonal 2D radiographs (sagittal and coronal views) to infer the 3D spinal pose.
  • Vertebral Shape Priors: The model incorporates two types of shape priors:
  • Spine Atlas: A global shape prior that enforces overall spinal curvature and consistency.
  • Vertebral Shape Prior: Local priors for individual vertebrae.
  • Hybrid Registration Approach: The reconstruction is framed as a two-stage registration process:
  • Affine Decoder: Estimates global affine transformations (scaling and rotation) for each vertebra, providing a rough alignment.
  • Deformable Decoder: A fully convolutional network that refines the shape by estimating local, voxel-wise deformations, capturing finer anatomical details.
  • Input Fusion: The network uses two encoders (one for sagittal, one for coronal views) that process image patches and their corresponding centroid annotations. These encoded features are then combined with global centroid coordinates and fed into the decoders.
  • Training on Synthetic Data: Due to the lack of 3D ground truth for standing spines, the model is trained on digitally reconstructed radiographs (DRRs) generated from CT scans, with supervision from the CT's 3D vertebral segmentation masks.

Datasets

  • Training Data:
    • A public dataset for lung nodule detection (800 chest CT scans).
  • An in-house dataset of 154 spinal CT scans.
  • Digitally Reconstructed Radiographs (DRRs) were generated from these CT scans.
  • Evaluation Data:
    • Clinical, standing radiographs from 30 patients.
  • For quantitative evaluation on clinical radiographs, CT scans from two patients were used, providing a comparison between the reconstructed 3D shape from radiographs and the patient's 3D CT segmentation (though in a different orientation).

Key Insights and Ideas

  • The Importance of "Anatomy-Awareness": Simply reconstructing individual vertebrae in 3D from 2D images isn't enough. Incorporating global spine shape priors (like the spine atlas) significantly improves the consistency and accuracy of the overall 3D reconstruction, preventing unnatural distortions.
  • Registration as a Deformable Template Problem: Framing the 3D reconstruction as a registration problem, where a template is deformed to match the target, is a powerful approach. This allows the model to leverage existing knowledge of vertebral shapes and learn how to adapt them to patient-specific anatomy.
  • The Challenge of Standing Spine Ground Truth: The lack of readily available 3D ground truth for standing spines is a major hurdle. The authors ingeniously address this by training on DRRs and developing a novel evaluation metric (nWESD) that is invariant to rotation, allowing comparison of reconstructed shapes to CT data even when the patient's posture differs.
  • Hybrid Affine and Deformable Registration: Combining a global affine transformation for initial alignment with a local deformable transformation for fine-tuning is a robust strategy for capturing both the overall spinal alignment and intricate vertebral details.

Key Results

  • Quantitative Performance on DRRs: The TransVert+ model achieved a Dice score of 0.95 on digitally reconstructed radiographs, indicating near-perfect 2D-to-3D domain translation.
  • Ablation Study: The TransVert+ model, which incorporates both affine and deformable decoders and shape priors, outperformed its predecessors:
  • Dice Score: TransVert+ achieved 0.9510, compared to TransVert (0.9426), DeformableTransVert (0.9405), and AffineTransVert (0.8847).
  • Hausdorff Distance: TransVert+ achieved 5.70 mm, significantly lower than TransVert (7.93 mm).
  • nWESD: TransVert+ achieved 0.1797, outperforming TransVert (0.2779).
  • Performance on Clinical Radiographs:
    • Qualitatively, the model successfully reconstructs patient-specific 3D spine models from clinical radiographs, capturing the natural curvature under weight-bearing.
  • Quantitatively, using the rotation-invariant nWESD metric, TransVert+ showed slightly better performance (mean nWESD of 0.13) compared to TransVert (0.15) when comparing reconstructed shapes to CT segmentations, though the difference was not statistically significant.
  • Vertebrae-wise Performance: The model generally performs well across all vertebrae, though performance drops slightly for lumbar vertebrae, likely due to their underrepresentation in the training data.

Limitations and Future Work

  • Lumbar Vertebrae Performance: The model's performance is slightly reduced for lumbar vertebrae, which the authors attribute to their underrepresentation in the training dataset. Future work could involve training with datasets containing more lumbar spine scans.
  • Orthogonality of Radiographs: Clinical radiographs may not always be perfectly orthogonal, whereas the DRRs used for training assume orthogonal projections. The authors suggest that incorporating randomness in projection angles during training could improve robustness.
  • Ground Truth for Standing Spines: The lack of direct 3D ground truth for standing spines remains a challenge for definitive validation. Acquiring more paired CT and radiograph data from the same patients in different postures could enrich validation efforts.
  • Data Augmentation: The authors suggest that data augmentation, particularly by introducing small randomness to projection angles in simulated radiographs, could help the model generalize better to clinical variations.

Summary updated

Abstract

An important factor for the development of spinal degeneration, pain and the outcome of spinal surgery is known to be the balance of the spine. It must be analyzed in an upright, standing position to ensure physiological loading conditions and visualize load-dependent deformations. Despite the complex 3D shape of the spine, this analysis is currently performed using 2D radiographs, as all frequently used 3D imaging techniques require the patient to be scanned in a prone position. To overcome this limitation, we propose a deep neural network to reconstruct the 3D spinal pose in an upright standing position, loaded naturally. Specifically, we propose a novel neural network architecture, which takes orthogonal 2D radiographs and infers the spine's 3D posture using vertebral shape priors. In this work, we define vertebral shape priors using an atlas and a spine shape prior, incorporating both into our proposed network architecture. We validate our architecture on digitally reconstructed radiographs, achieving a 3D reconstruction Dice of 0.95, indicating an almost perfect 2D-to-3D domain translation. Validating the reconstruction accuracy of a 3D standing spine on real data is infeasible due to the lack of a valid ground truth. Hence, we design a novel experiment for this purpose, using an orientation invariant distance metric, to evaluate our model's ability to synthesize full-3D, upright, and patient-specific spine models. We compare the synthesized spine shapes from clinical upright standing radiographs to the same patient's 3D spinal posture in the prone position from CT.
BibTeX
@inproceedings{bayat2022anatomy,
  author = {Amirhossein Bayat and Danielle F. Pace and Anjany Sekuboyina and Christian Payer and Darko Štern and Martin Urschler and Jan S. Kirschke and Bjoern Menze},
  title = {{Anatomy-Aware Inference of the 3D Standing Spine Posture from 2D Radiographs}},
  booktitle = {Tomography},
  year = {2022},
  doi = {10.3390/tomography8010039},
}
Figure