Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging

Vivek Gopalakrishnan, Polina Golland

TL;DR

This paper introduces a novel, fully differentiable DRR (Digitally Reconstructed Radiograph) generator that significantly speeds up image registration for intraoperative imaging. By reformulating a standard ray-tracing algorithm as vectorized tensor operations, they achieve near real-time performance and enable gradient-based optimization, which was previously too slow for clinical use.

What Problem Does This Paper Address?

Intraoperative imaging, like during surgery, often relies on aligning pre-operative 3D scans (like CT) with live 2D X-ray images to guide procedures. Digitally Reconstructed Radiographs (DRRs) are simulated X-rays generated from 3D volumes, crucial for this alignment. However, generating DRRs quickly enough for real-time intraoperative use has been a major bottleneck. Furthermore, optimizing the alignment often requires calculating how changes in the 3D pose affect the DRR (i.e., its gradients). Existing DRR generators, often implemented in low-level languages like CUDA/C++, make it difficult and computationally expensive to obtain these gradients, forcing researchers to rely on slower, gradient-free optimization methods. This paper aims to bridge this gap by creating a DRR generator that is both fast and fully differentiable.

Methods

The core innovation lies in reformulating Siddon's method, a popular ray-tracing algorithm for DRR generation, into a series of vectorized tensor operations. This is implemented in PyTorch, a deep learning framework known for its automatic differentiation capabilities.

  • Vectorized Siddon's Method: Instead of tracing each ray individually, the authors treat the entire detector plane and the 3D volume as tensors. They compute ray-plane intersections and voxel contributions in a highly parallelized manner, leveraging GPU acceleration.
  • Auto-Differentiability: By performing all operations within PyTorch, the DRR generation process becomes a differentiable function. This means that the framework can automatically compute the gradients of any loss function (e.g., image similarity) with respect to the DRR's generating parameters (like the X-ray source position and orientation).
  • Comparison to Existing Methods: They compare their approach to a CPU-based implementation (Plastimatch) and a partially vectorized GPU implementation of an iterative method (Siddon-Jacobs).

Datasets

The paper uses a reference chest CT scan from Slicer3D [6] for evaluation. No new datasets or benchmarks are introduced; the focus is on improving the DRR generation process itself.

Key Insights and Ideas

  • Gradient-Based Optimization is Feasible: The paper demonstrates that for slice-to-volume registration, the loss landscape is often convex around the optimal solution. This makes gradient-based optimization a viable and much faster alternative to gradient-free methods, but only if the gradients can be efficiently computed.
  • Vectorization Unlocks Differentiability: By moving away from iterative, low-level implementations and embracing vectorized tensor operations within a framework like PyTorch, they achieve both speed and automatic differentiation.
  • Real-Time Potential: The achieved rendering speeds are comparable to state-of-the-art GPU-accelerated DRR generators, opening the door for real-time applications in intraoperative settings.
  • Convexity of Loss Landscapes: The finding that ZNCC loss landscapes are locally convex for this registration problem is crucial, as it justifies the use of gradient descent and highlights the potential of their differentiable DRR generator.

Key Results

  • Speed: Their vectorized GPU implementation achieves rendering speeds equivalent to state-of-the-art CUDA/C++ implementations, with timings as low as 72.7 ms for a 200x200 DRR on an NVIDIA GeForce RTX 2080 Ti. This is a significant improvement over CPU-based methods (e.g., 424 ms for a 200x200 DRR).
  • Accuracy: The generated DRRs show high similarity to existing methods, with an average RMSE of (8.3±1.9)×10⁻⁴ compared to Plastimatch.
  • Gradient Efficiency: Automatically computed gradients are an order of magnitude faster to generate (35.1 ms) compared to finite differences (400.5 ms).
  • Slice-to-Volume Registration: They successfully demonstrate the use of their differentiable DRR generator for a 6 DoF slice-to-volume registration problem, achieving convergence in an average of 65.48 iterations (1.92 seconds).

Limitations and Future Work

  • Realistic DRRs: The current DRR model does not account for scattering effects, which are present in real X-ray systems. This limits the realism of the generated images.
  • Future Work: The authors plan to investigate fusing their method with DeepDRR, a deep learning framework that estimates scattering effects, to produce DRRs that are simultaneously realistic, fast, and differentiable. They also aim to explore using their differentiable DRRs for pre-training deep learning models for specific intraoperative imaging tasks, potentially leading to even faster and more accurate solutions.

Summary updated

Abstract

The use of digitally reconstructed radiographs (DRRs) to solve inverse problems such as slice-to-volume registration and 3D reconstruction is well-studied in preoperative settings. In intraoperative imaging, the utility of DRRs is limited by the challenges in generating them in real-time and supporting optimization procedures that rely on repeated DRR synthesis. While immense progress has been made in accelerating the generation of DRRs through algorithmic refinements and GPU implementations, DRR-based optimization remains slow because most DRR generators do not offer a straightforward way to obtain gradients with respect to the imaging parameters. To make DRRs interoperable with gradient-based optimization and deep learning frameworks, we have reformulated Siddon’s method, the most popular ray-tracing algorithm used in DRR generation, as a series of vectorized tensor operations. We implemented this vectorized version of Siddon’s method in PyTorch, taking advantage of the library’s strong automatic differentiation engine to make this DRR generator fully differentiable with respect to its parameters. Additionally, using GPU-accelerated tensor computation enables our vectorized implementation to achieve rendering speeds equivalent to state-of-the-art DRR generators implemented in CUDA and C++. We illustrate the resulting method in the context of slice-to-volume registration. Moreover, our simulations suggest that the loss landscapes for the slice-to-volume registration problem are convex in the neighborhood of the optimal solution, and gradient-based registration promises a much faster solution than prevailing gradient-free optimization strategies. The proposed DRR generator enables fast computer vision algorithms to support image guidance in minimally invasive procedures. Our implementation is publically available at https://github.com/v715/DiffDRR .
BibTeX
@inproceedings{gopalakrishnan2022fast,
  author = {Vivek Gopalakrishnan and Polina Golland},
  title = {{Fast Auto-Differentiable Digitally Reconstructed Radiographs for Solving Inverse Problems in Intraoperative Imaging}},
  booktitle = {CLIP @ MICCAI},
  year = {2022},
}
Figure