FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks

David Bauer, Qi Wu, Kwan-Liu Ma

Abstract

Volume data is found in many important scientific and engineering applications. Rendering this data for visualization at high quality and interactive rates for demanding applications such as virtual reality is still not easily achievable even using professional-grade hardware. We introduce FoVolNet -- a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network. Foveated rendering is a technique that prioritizes rendering computations around the user's focal point. This approach leverages properties of the human visual system, thereby saving computational resources when rendering data in the periphery of the user's field of vision. Our reconstruction network combines direct and kernel prediction methods to produce fast, stable, and perceptually convincing output. With a slim design and the use of quantization, our method outperforms state-of-the-art neural reconstruction techniques in both end-to-end frame times and visual quality. We conduct extensive evaluations of the system's rendering performance, inference speed, and perceptual properties, and we provide comparisons to competing neural image reconstruction techniques. Our test results show that FoVolNet consistently achieves significant time saving over conventional rendering while preserving perceptual quality.

Introduction

Volume rendering is a cornerstone of scientific and medical visualization, essential for interpreting complex data ranging from medical CT scans to astrophysical simulations. However, high-fidelity rendering—incorporating effects like global illumination—imposes a heavy computational burden. This creates a "tight coupling" between visual quality and performance, often making interactive frame rates unreachable for demanding applications like Virtual Reality (VR).

FoVolNet addresses this challenge by leveraging the biological principles of the human visual system. By concentrating computational effort on the user's focal point (the fovea) and reducing detail in the periphery, FoVolNet breaks the traditional quality-cost trade-off, enabling high-performance visualization on standard consumer hardware.

Methodology

The FoVolNet pipeline is divided into two primary stages: foveated sampling and neural reconstruction.

Foveated Sparse Rendering

Instead of rendering every pixel in a standard frame, the system calculates only a small subset. It utilizes Spatio-Temporal Blue Noise (STBN) to generate a sampling mask that is dense at the fovea and increasingly sparse toward the edges. To optimize GPU utilization, a Stream Compaction algorithm packs these active pixels into a dense buffer for rendering before they are projected back to their original screen-space coordinates.

The foveated sampling process: Spatio-Temporal Blue Noise is used to create a mask that prioritizes pixel density at the user's focal point.

The W-Net Reconstruction Network

The core of the system is a hybrid neural network based on the W-Net architecture, consisting of two sequential U-Nets:

  • Network D (Direct Prediction): Performs a coarse reconstruction to fill large gaps between sparse samples.
  • Network K (Kernel Prediction): Refines the output of Network D by predicting local convolutional filters to sharpen details and eliminate artifacts.
  • Temporal Recurrence: The network utilizes recurrent connections to maintain temporal stability, effectively reducing the flickering or "crawling" artifacts common in sparse neural reconstruction.

Evaluation and Results

FoVolNet was evaluated across diverse high-resolution datasets, including biological micro-CT scans (Chameleon), medical data (Skull), and engineering models (Mechanical Hand). To test generalization, the Skull and Chameleon datasets were reserved exclusively for evaluation.

Performance and Quality

FoVolNet achieved end-to-end rendering speeds averaging 3.28x faster than conventional raymarching. The stream compaction method proved nearly 10x more efficient than naive pixel-skipping. In head-to-head comparisons with previous state-of-the-art methods like DeepFovea, FoVolNet consistently delivered superior PSNR and SSIM scores.

Visual comparison across three datasets (Mechanical Hand, Skull, and Chameleon) demonstrating FoVolNet's ability to reconstruct high-fidelity images with significant speedups.

Key Insights

  • Hybrid Prediction: Combining direct prediction (for structure) and kernel prediction (for detail preservation) is essential for high-quality results.
  • Blue Noise Advantage: Blue noise sampling prevents the formation of low-frequency clusters, allowing the neural network to reconstruct images without blotchy artifacts.
  • Precision Optimization: Utilizing FP16 half-precision during inference provides a significant speed boost with negligible impact on visual quality, though further reduction to INT8 can introduce color banding.

Conclusion and Future Work

FoVolNet demonstrates that foveated neural reconstruction can significantly accelerate volume rendering while maintaining perceptual fidelity. While the system is highly effective, future work aims to address high-frequency flickering in extremely dense regions—potentially through Generative Adversarial Network (GAN) losses—and explore "Neural Adaptive Sampling" to dynamically allocate rendering resources to areas of high geometric complexity. The pipeline is also a promising candidate for stabilizing frame rates in computationally expensive Monte Carlo path tracing.

Summary updated

Abstract

Volume data is found in many important scientific and engineering applications. Rendering this data for visualization at high quality and interactive rates for demanding applications such as virtual reality is still not easily achievable even using professional-grade hardware. We introduce FoVolNet -- a method to significantly increase the performance of volume data visualization. We develop a cost-effective foveated rendering pipeline that sparsely samples a volume around a focal point and reconstructs the full-frame using a deep neural network. Foveated rendering is a technique that prioritizes rendering computations around the user's focal point. This approach leverages properties of the human visual system, thereby saving computational resources when rendering data in the periphery of the user's field of vision. Our reconstruction network combines direct and kernel prediction methods to produce fast, stable, and perceptually convincing output. With a slim design and the use of quantization, our method outperforms state-of-the-art neural reconstruction techniques in both end-to-end frame times and visual quality. We conduct extensive evaluations of the system's rendering performance, inference speed, and perceptual properties, and we provide comparisons to competing neural image reconstruction techniques. Our test results show that FoVolNet consistently achieves significant time saving over conventional rendering while preserving perceptual quality.
BibTeX
@article{bauer2022fovolnet,
  author = {David Bauer and Qi Wu and Kwan-Liu Ma},
  title = {{FoVolNet: Fast Volume Rendering using Foveated Deep Neural Networks}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2022},
  doi = {10.1109/tvcg.2022.3209498},
}
Figure