Efficient Raycasting of Volumetric Depth Images for Remote Visualization of Large Volumes at High Frame Rates

Aryaman Gupta, Ulrik Günther, Pietro Incardona, Guido Reina, Steffen Frey, Stefan Gumhold, Ivo F. Sbalzarini

TL;DR

This paper introduces a high-performance raycasting algorithm for Volumetric Depth Images (VDIs), enabling interactive remote visualization of massive scientific datasets at HD resolutions. By transforming complex frustum intersections into simplified cuboid math in Normalized Device Coordinate (NDC) space and leveraging spatial coherence, the authors achieve frame rates up to an order of magnitude faster than previous VDI techniques.

What Problem Does This Paper Address?

Interactive Direct Volume Rendering (DVR) is a cornerstone of scientific data analysis, but it faces two major hurdles: the sheer size of modern datasets (often gigabytes) and the latency inherent in remote visualization.

Volumetric Depth Images (VDIs) were proposed to solve this by "baking" a view-dependent version of the volume into a compact representation of color and opacity segments. This allows a client to perform local re-rendering for small viewpoint changes, decoupling interaction from expensive server-side calculations. However, previous VDI rendering methods were either geometry-heavy (creating millions of triangles) or computationally inefficient, preventing them from reaching smooth frame rates at Full HD resolutions.

Methods

The authors overhaul the VDI pipeline with several key innovations:

  • NDC Space Raycasting: In world space, VDI segments are irregular pyramidal frustums, making intersection tests mathematically expensive. The authors perform raycasting in the Normalized Device Coordinate (NDC) space of the original viewpoint. In this space, the frustums become axis-aligned cuboids, allowing the use of the highly efficient "fast voxel traversal" algorithm.
  • Spatial Smoothness Optimization: Adjacent rays in a VDI are likely to intersect similar segments. The algorithm uses the intersection index from a neighboring ray as a "seed" to start its search, reducing memory accesses and yielding up to a 40% speedup over standard binary search.
  • Ray-Adaptive Generation: Generating a VDI usually requires manual tuning of a sensitivity parameter ($\gamma$) to decide where to start new segments. The authors implement an automatic bisection search that finds the optimal $\gamma$ for each individual ray, maximizing visual quality within a fixed segment budget.
  • Empty-Space Skipping: They supplement the VDI with a 3D grid acceleration structure. If a grid cell contains zero segments, the ray simply jumps over that region.
  • Dynamic Subsampling: To maintain high frame rates during rapid camera movements, a Proportional-Integral (PI) controller dynamically adjusts image-space and ray-space sampling density.
The structure of a VDI and the coordinate transformations involved in rendering. Note how the irregular frustums in world space (a) become regular cuboids in NDC space (b).

Datasets

The algorithm was evaluated on four diverse volume datasets:

  1. Engine: A 256³ CT scan of an engine block.
  2. Kingsnake: A sparse 1024x1024x795 X-ray CT scan of a snake egg.
  3. Rayleigh-Taylor: A dense 1024³ fluid simulation.
  4. Richtmyer-Meshkov: A massive 2048x2048x1920 entropy field from a fluid instability simulation.

Key Insights and Ideas

  • The Power of Perspective: The most significant "aha" moment is realizing that the perspective projection used to generate the VDI can be inverted to simplify the rendering math. Turning a frustum into a box (via NDC) is a classic graphics trick that perfectly fits the VDI data structure.
  • Anisotropy as a Feature: VDI rendering is naturally faster when the new viewpoint is close to the original one because the rays "march" along the compressed dimension of the data.
  • Decoupling is Key for Remote Viz: By rendering the VDI locally on the client (even a less powerful one like a workstation with an AMD RX 5700XT), the user experiences zero network lag for small rotations and zooms, even if the server is still crunching the next full volume update.
Figure 1: Visual comparison of VDI rendering quality with direct volume rendering (DVR) for the Kingsnake (top), Rayleigh- Taylor (middle), and Richtmyer-Meshkov (bottom) datasets. All frame rates are reported without empty-space skipping for comparability. Image quality metrics are computed w.r.t. the DVR image at the same angle.

Key Results

  • Massive Speedups: The new raycasting method is often 10x faster than previous rasterization-based VDI techniques.
  • Outperforming DVR: Compared to standard Direct Volume Rendering, the VDI approach is 4.5x to 24.5x faster for standard HD resolutions at small viewpoint deviations.
  • High Fidelity: Image quality remains excellent, with SSIM (Structural Similarity Index) values typically staying above 0.9 even for viewpoint deviations up to 40 degrees.
  • Predictable Memory: Unlike geometry-based methods, the memory footprint is constant and predictable (e.g., ~480 MB for a 1024² VDI), regardless of data density.

Limitations and Future Work

  • Lighting Constraints: The current VDI structure stores pre-classified color and opacity. This means it cannot easily handle directional lighting effects like specular highlights, which change based on the light source and view angle.
  • Manual Tuning: While image-space subsampling is automated via a PI controller, subsampling along the ray still requires some manual activation/tuning in the current implementation.
  • Predictive Streaming: The authors suggest that future versions could use camera motion prediction to speculatively generate VDIs for where the user is going to look, further reducing perceived latency.

The implementation is available as an open-source extension to the scenery visualization library.

Summary updated

Abstract

We present an efficient raycasting algorithm for rendering Volumetric Depth Images (VDIs), and we show how it can be used in a remote visualization setting with VDIs generated and streamed from a remote server. VDIs are compact view-dependent volume representations that enable interactive visualization of large volumes at high frame rates by decoupling viewpoint changes from expensive rendering calculations. However, current rendering approaches for VDIs struggle with achieving interactive frame rates at high image resolutions. Here, we exploit the properties of perspective projection to simplify intersections of rays with the view-dependent frustums in a VDI and leverage spatial smoothness in the volume data to minimize memory accesses. Benchmarks show that responsive frame rates can be achieved close to the viewpoint of generation for HD display resolutions, providing high-fidelity approximate renderings of Gigabyte-sized volumes. We also propose a method to subsample the VDI for preview rendering, maintaining high frame rates even for large viewpoint deviations. We provide our implementation as an extension of an established open-source visualization library.
BibTeX
@inproceedings{gupta2023efficient,
  author = {Aryaman Gupta and Ulrik Günther and Pietro Incardona and Guido Reina and Steffen Frey and Stefan Gumhold and Ivo F. Sbalzarini},
  title = {{Efficient Raycasting of Volumetric Depth Images for Remote Visualization of Large Volumes at High Frame Rates}},
  booktitle = {Pacificvis},
  year = {2023},
  doi = {10.1109/pacificvis56936.2023.00014},
}
Figure