UltraVoxelGS: Voxel-First Feed-Forward Gaussian Splatting for 3D Ultrasound Reconstruction

Yue Wang, Haoxin Chen, Xiaoqing GuoarXiv 2026 · arXiv

TL;DR

UltraVoxelGS is the first feed-forward Gaussian Splatting framework designed specifically for 3D ultrasound reconstruction, slashing processing times from 20 minutes to just 10 seconds. By combining a "voxel-first" architecture with a physics-aware rendering strategy, it enables high-fidelity 3D visualization and real-time spatial feedback for sonographers during 2D scanning.

What Problem Does This Paper Address?

Freehand 2D ultrasound is the most common diagnostic imaging modality, but it requires significant "mental gymnastics" from sonographers, who must reconstruct 3D anatomy in their heads from a sequence of 2D slices. This often leads to incomplete coverage and missed pathology.

While 3D reconstruction methods exist, they face a "speed vs. quality" deadlock. Traditional interpolation is fast but produces blurry, physically inaccurate results. Modern neural methods like Neural Radiance Fields (NeRF) or Gaussian Splatting (GS) produce stunning images but require iterative, per-scene optimization. Waiting 10–20 minutes for a reconstruction is a non-starter for a live clinical exam.

Furthermore, ultrasound isn't like natural photography. Standard feed-forward GS models assume dense overlap between images and consistent lighting. Ultrasound slices are incredibly thin (sharing almost no overlap) and exhibit "acoustic anisotropy"—the same tissue looks different depending on the angle of the probe due to how sound waves propagate and attenuate.

Figure 1: Clinical workflow. A sequence of posed 2D ultrasound slices are reconstructed into a 3D representation, from which novel views are rendered to guide the operator toward unseen regions.

Methods

The authors introduce UltraVoxelGS, which bypasses the optimization bottleneck using a feed-forward design. Its architecture rests on three pillars:

  1. Voxel-First Reformulation: Instead of predicting Gaussians directly from 2D pixels (which fails due to sparse overlap), the method "lifts" 2D slices into a unified 3D voxel grid based on probe poses. This creates a shared spatial context that a Transformer-based network can then use to predict Gaussian parameters.
  2. Ultrasound-Aware Rendering: The model doesn't just predict color; it factorizes appearance into anisotropic reflection (how sound bounces off tissue) and cumulative attenuation (how the signal weakens as it travels deeper). This respects the actual physics of acoustic imaging.
  3. Render-time Appearance Adaptation (RAA): To recover fine details lost during the feed-forward pass, the system performs a lightweight refinement at render-time. It uses a nearby observed slice as an "anchor" to sharpen the target view without changing the underlying 3D geometry.
Figure 2: Optical feed-forward GS paradigm: (a) Natural images provide dense cross-view overlap and photometric consistency. (b) Existing methods predict pixel-aligned primitives from individual views. Our ultrasound-specific feed-forward GS paradigm: (c) Ultrasound slices share minimal intersection and exhibit view-dependent intensity. (d) We lift posed slices into a unified voxel support and predict Gaussians from the shared voxel support.

Datasets

The model was evaluated on two distinct datasets:

  • Dataset A (Real-world): Robotic-assisted lumbar spine ultrasound scans from 63 volunteers, featuring over 3,000 frames per scan with synchronized probe tracking.
  • Dataset B (Simulated): 100 simulated liver ultrasound sequences generated from abdominal CT volumes, providing a ground-truth reference for anatomical consistency.

Key Insights and Ideas

  • Decoupling Representation from Input Count: By using a fixed-size voxel support, the computational cost doesn't explode when processing the thousands of slices typical in a freehand scan.
  • Thin-Slab Locality: The authors realized that to reconstruct a specific slice, you only need to look at the "acoustic slab" (the local volume) around it. This allows for local-to-global assembly, making the model scalable to very long sequences.
  • Physics as a Prior: Explicitly modeling attenuation and reflection allows the model to handle the view-dependent nature of ultrasound much better than generic "black-box" neural networks.

Key Results

UltraVoxelGS outperforms both interpolation-based and optimization-based baselines in both speed and quality:

  • Speed: It reduces reconstruction time from 10–20 minutes to ~10 seconds, a 35× speedup that brings the technology within reach of real-time clinical use.
  • Accuracy: On real-world data, it achieved a PSNR of 24.29, significantly higher than standard 3D Gaussian Splatting (22.55) and NeRF-based methods (18.49).
  • Scalability: The method scales gracefully to sequences exceeding 1,000 frames, maintaining high quality while only mildly increasing memory usage.
  • Consistency: In simulated tests, it showed superior 3D anatomical consistency (measured via Dice score and IoU) compared to all other methods.

Limitations and Future Work

While a major leap forward, the authors note several areas for improvement:

  • Tracking Dependency: The method relies on accurate probe tracking. Errors in the recorded pose will degrade the 3D reconstruction.
  • Simplified Physics: The current model focuses on reflection and attenuation but does not yet account for complex effects like refraction, speed-of-sound variations, or speckle decorrelation.
  • Clinical Validation: While the image metrics (PSNR/SSIM) are strong, the authors emphasize that further study is needed to establish how this improved visualization translates to actual clinical diagnostic performance.

Future work may involve incorporating richer acoustic effects and developing explicit uncertainty estimation to help guide sonographers toward regions that need more scanning.

Summary updated

Abstract

We present UltraVoxelGS, a 3D ultrasound reconstruction method that achieves both fast reconstruction and physically faithful rendering for the first time, bringing live spatial feedback during 2D ultrasound scanning within reach. This is made possible by a feed-forward design that predicts a complete Gaussian scene representation from posed ultrasound images directly, entirely bypassing the per-scene optimization bottleneck that has confined prior methods to offline use. Realising such feed-forward prediction in this setting is non-trivial: unlike natural images, ultrasound slices share at most one-dimensional intersections and exhibit view-dependent appearance due to acoustic propagation and attenuation, violating the dense overlap and photometric consistency assumed by existing feed-forward Gaussian methods. Hence, to achieve fast reconstruction, we introduce a feed-forward voxel-to-Gaussian pass that decouples representation size from input count by lifting all posed slices into a fixed-size voxel support and predicting Gaussian parameters directly. For faithful rendering, we propose ultrasound-aware appearance rendering strategy factorize reflection from cumulative attenuation during training and exploit appearance continuity among spatially adjacent slices for efficient Render-time Appearance Adaptation. On both real-world and simulated datasets, UltraVoxelGS achieves best PSNR/SSIM compared with per-scene optimization baselines while reducing inference from 10-20 minutes to approximately 10 seconds (over 35 times faster) and scales to sequences exceeding 1,000 frames. Code is available at https://github.com/YuebyYue/UltraVoxelGS-camera-ready.
BibTeX
@misc{wang2026ultravoxelgs,
  author = {Yue Wang and Haoxin Chen and Xiaoqing Guo},
  title = {{UltraVoxelGS: Voxel-First Feed-Forward Gaussian Splatting for 3D Ultrasound Reconstruction}},
  year = {2026},
  eprint = {2611.11951},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11951},
}
Figure