FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks

David Bauer, Qi Wu, Kwan‐Liu Ma

Introduction

Volume data visualization is a cornerstone of modern scientific and medical research, enabling the exploration of complex internal structures in CT scans, fluid dynamics simulations, and astrophysical models. However, high-quality volume rendering—particularly using advanced techniques like path tracing—remains computationally prohibitive for real-time applications. This is especially true for Virtual Reality (VR), where high frame rates and low latency are non-negotiable for a comfortable user experience.

FoVolNet addresses this challenge by introducing a foveated rendering pipeline that decouples visual quality from total pixel count. By prioritizing computational resources in the user's central field of vision (the fovea) and utilizing a sophisticated deep neural network for peripheral reconstruction, FoVolNet achieves significant performance gains without sacrificing perceived image quality.

The FoVolNet Methodology

The system consists of two primary components: a sparse foveated renderer and a hybrid neural reconstruction network.

Sparse Foveated Rendering

Traditional rendering processes every pixel on the screen. FoVolNet instead employs a sparse sampling strategy based on the human visual system. Using Spatio-Temporal Blue Noise (STBN), the system selects a subset of pixels to render. Sampling density is highest at the gaze point and decreases towards the periphery.

A key technical innovation in this stage is the use of stream compaction. On modern GPUs, simply skipping pixels in a shader often fails to save time because threads are executed in "warps"; if even one pixel in a warp needs rendering, the entire warp remains active. FoVolNet's stream compaction algorithm packs sparse samples into a dense buffer, ensuring that GPU threads are fully utilized and translating theoretical sparsity into actual wall-clock speedups.

Hybrid Neural Reconstruction

The sparse output is then passed to a specialized "W-Net" architecture for reconstruction. This network employs a hybrid approach, combining two distinct strategies:

  1. Direct Prediction (Network D): This sub-network performs a coarse reconstruction to fill the large gaps in the sparse peripheral data. While effective at establishing structure, direct prediction can often result in blurry outputs.
  2. Kernel Prediction (Network K): To restore fine detail, Network K predicts per-pixel convolutional kernels. These kernels are applied to the input and the features from Network D to sharpen the image and remove artifacts.

To ensure temporal stability—preventing the "shimmering" common in AI-upscaled video—the network incorporates recurrent connections and is trained using an optical flow loss. This allows the model to leverage information from previous frames to maintain consistency over time.

Performance and Results

FoVolNet was rigorously tested on diverse datasets, including medical CT scans (human skull, chameleon) and scientific simulations (supernova, fluid vortices). The results demonstrate a significant leap over existing methods:

  • Computational Efficiency: FoVolNet achieved an average end-to-end speedup of 3.28x compared to conventional full-resolution raymarching. When compared to previous state-of-the-art neural upscalers like DeepFovea, FoVolNet provided speedups up to 25x at high compression ratios.
  • Visual Fidelity: Despite the sparse input, the reconstructed images maintained high structural similarity (SSIM) and peak signal-to-noise ratio (PSNR).
  • Perceptual Quality: Using the Just-Objectionable-Difference (JOD) metric, FoVolNet scored near the maximum value of 10. This indicates that the reconstructed frames are virtually indistinguishable from ground-truth references to a human observer.

Conclusion and Future Directions

FoVolNet represents a significant advancement in real-time volume visualization. By intelligently combining the principles of human perception with efficient GPU algorithms and hybrid deep learning, it breaks the traditional bottleneck of volume rendering.

While the current implementation focuses on raymarching, the framework is extensible to even more demanding techniques like Monte Carlo volume path tracing. Future work may explore "Neural Adaptive Sampling," where the network dynamically determines sampling density based on the local complexity of the volumetric data itself, further optimizing the balance between performance and quality.

Summary updated

Abstract

Volume data is found in many important scientific and engineering applications. Rendering this data for visualization at high quality and interactive rates for demanding applications such as virtual reality is still not easily achievable even using professional-grade hardware. We introduce FoVolNet-a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network. Foveated rendering is a technique that prioritizes rendering computations around the user's focal point. This approach leverages properties of the human visual system, thereby saving computational resources when rendering data in the periphery of the user's field of vision. Our reconstruction network combines direct and kernel prediction methods to produce fast, stable, and perceptually convincing output. With a slim design and the use of quantization, our method outperforms state-of-the-art neural reconstruction techniques in both end-to-end frame times and visual quality. We conduct extensive evaluations of the system's rendering performance, inference speed, and perceptual properties, and we provide comparisons to competing neural image reconstruction techniques. Our test results show that FoVolNet consistently achieves significant time saving over conventional rendering while preserving perceptual quality.
BibTeX
@article{bauer2022fovolnet,
  author = {David Bauer and Qi Wu and Kwan‐Liu Ma},
  title = {{FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2022},
  doi = {10.1109/tvcg.2022.3209498},
}
Figure