Raysemble: Metric 3D Bone Reconstruction from a Single X-ray

Xiaoyang Li, Xinghao Yan, Yuan Chai

TL;DR

Raysemble is a new deep-learning framework that reconstructs eight major bones (the lumbar spine and pelvis) in metric 3D space from just a single 2D X-ray. By cleverly decoupling the prediction of a bone's normalized shape from its physical placement, it achieves state-of-the-art accuracy, significantly outperforming existing volume-rendering and shape-model approaches.

What Problem Does This Paper Address?

Reconstructing 3D anatomy from a single 2D projection is a classic "ill-posed" problem in medical imaging. When you look at an anteroposterior (AP) X-ray, several factors obscure the true 3D geometry:

  • Depth Ambiguity: Different 3D positions can result in nearly identical 2D outlines.
  • Projection Magnification: Because X-rays are cone-beam projections, an object's apparent size in the image depends on its distance from the source.
  • Anatomical Overlap: Bones like the sacrum and hips overlap in 2D, making it hard to distinguish individual boundaries.

Existing methods often struggle to recover the actual physical size (metric reconstruction) or fail to capture fine local details while maintaining global anatomical consistency.

Figure 1. From one AP radiograph to eight bones. (a) The reconstruction task. (b) Depth ambiguity, projection magnification, and anatomical overlap obscure 3D geometry. (c) Raysemble separates normalized bone shape from physical placement, then combines both through x = c + sq in a shared right–anterior–superior (RAS) coordinate system.

Methods

The core innovation of Raysemble is its dual-branch architecture that separates "what the bone looks like" from "where it is in the room."

  1. Anatomical Initialization: The system starts by using a 2D segmenter to create probability maps. These maps help define a Region of Interest (ROI) for each of the eight bones (L1–L5, sacrum, and two hip bones).
  2. Shape Branch: This branch predicts the occupancy of a bone on a normalized grid. To get the best of both worlds, it samples features from a Full-frame CNN (for global anatomical context) and a Regional CNN (for high-resolution local detail).
  3. Placement Branch: Instead of just guessing a 3D volume, this branch predicts the bone's depth and uses known camera geometry to back-project the 2D center into 3D space. It also predicts a scale factor to correct for magnification.
  4. RAS Assembly: Finally, the predicted normalized shapes are resampled onto a common "Right-Anterior-Superior" (RAS) coordinate system in millimeters. This ensures the final output is a metric 3D model where bones are correctly sized and positioned relative to one another.
The Raysemble pipeline architecture

Datasets

The authors trained and evaluated Raysemble using:

  • CADS (CT-TRI and AbdomenCT-1K): A large dataset of 419 CT scans from 225 subjects.
  • Digitally Reconstructed Radiographs (DRRs): Since paired X-ray/CT data is rare, they used the DiffDRR tool to simulate realistic AP radiographs from the CT volumes.
  • Real Clinical Data: For exploratory validation, they tested the model on two real pelvic AP radiographs from a hospital, comparing the results to subsequent CT scans of the same patients.

Key Insights and Ideas

  • Separation of Concerns: By predicting a "canonical" bone shape and a separate placement vector, the model doesn't have to learn the infinite variations of how a bone might appear at different depths and angles simultaneously.
  • Ray Lifting: The method uses "projection-aligned features." For any 3D query point, it projects that point back onto the 2D image to grab the relevant local and global features. This keeps the 3D prediction grounded in the 2D evidence.
  • Metric Matters: Most research in this area focuses on "shape similarity" (like Dice scores). Raysemble emphasizes "metric reconstruction," aiming for accuracy in millimeters, which is far more useful for clinical applications like surgical planning.

Key Results

Raysemble was compared against 15 other baseline methods, including popular architectures like VolumeNeRF and various U-Net adaptations.

  • Superior Accuracy: It achieved a 65.24% Dice score, a 9.09 mm HD95 (95th percentile Hausdorff distance), and a 3.32 mm ASSD (Average Symmetric Surface Distance).
  • Significant Gains: Compared to its closest competitor, VolumeNeRF, it improved the Dice score by 5.86 percentage points using identical inputs.
  • Robustness: Qualitative results (Figure 3) show that Raysemble produces much smoother and more anatomically plausible bone surfaces than voxel-based or direct-regression baselines, which often produce "noisy" or disconnected volumes.
Qualitative comparison of Raysemble against other methods

Limitations and Future Work

While impressive, the authors note several areas for improvement:

  • Camera Calibration: The model currently relies on knowing the camera geometry (source-to-detector distance, etc.). In some clinical settings, this metadata might be missing or inaccurate.
  • Truncation: If a bone is partially cut off at the edge of the X-ray, the ROI-based feature sampling can be degraded.
  • Single View: While it performs well with one image, adding a lateral (side) view would likely resolve even more depth ambiguity.
  • Pathology: The current model is trained on relatively "normal" anatomy; its performance on severe spinal deformities or fractures remains an open question for future research.

Summary updated

Abstract

Raysemble reconstructs eight lumbar and pelvic bones from a single X-ray by separating shape and placement. Each 3D query samples regional and global features at its image projection. A shared decoder predicts normalized bone shapes. Predicted depth and camera geometry determine 3D centers and scales for resampling these shapes onto a common grid in millimeters. Evaluation uses 23 held-out patients with 43 CT scans, each generating one simulated anteroposterior radiograph. Without rigid alignment to CT,Raysemble leads 15 evaluated methods, averaging 65.24% Dice, 9.09 mm HD95, and 3.32 mm ASSD over three random seeds. With identical inputs, it improves Dice over VolumeNeRF by 5.86 percentage points.
BibTeX
@misc{li2026raysemble,
  author = {Xiaoyang Li and Xinghao Yan and Yuan Chai},
  title = {{Raysemble: Metric 3D Bone Reconstruction from a Single X-ray}},
  year = {2026},
  eprint = {2611.16869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.16869},
}

Loading related content...

Figure