TL;DR
This paper introduces RAKD, a novel framework for real-time Monte Carlo (MC) ray tracing denoising that uses knowledge distillation to create a lightweight, high-quality denoiser. By combining render-aware initialization, adversarial learning, and an auxiliary dataset, RAKD achieves state-of-the-art denoising performance while meeting real-time constraints.
What Problem Does This Paper Address?
Real-time ray tracing (RTRT) offers photorealistic graphics but is computationally expensive, forcing the use of very low sampling rates (1-4 samples per pixel). This results in noisy images that degrade visual quality. While neural networks have shown great promise in denoising Monte Carlo (MC) rendered images, existing methods often require significant computational resources and memory, making them unsuitable for real-time applications on resource-limited devices. Existing lightweight neural models, while efficient, often sacrifice image quality. This paper aims to bridge this gap by developing a neural denoiser that is both computationally efficient for real-time use and capable of producing high-quality, noise-free images.
Methods
The core of this work is the Render-Aware Knowledge Distillation (RAKD) framework. RAKD employs a "teacher-student" paradigm where a large, high-performance teacher network distills its knowledge into a smaller, efficient student network. Key components of RAKD include:
- Local-to-Global Knowledge Distillation: This module ensures that knowledge is transferred at both the pixel level (local) and the image level (global).
- Local Distillation: A pixel-wise loss is computed between the denoised outputs of the teacher and student models.
- Global Distillation: A Generative Adversarial Network (GAN) is used. The student model acts as the generator, producing "fake" denoised images, while the teacher's outputs are treated as "real." A discriminator is trained to distinguish between them, pushing the student to mimic the teacher's overall image quality. Wasserstein distance is used for stable GAN training.
- Render-Aware Knowledge Initialization: Instead of random initialization, the student network's weights are initialized based on the teacher network's weights. This leverages the teacher's learned understanding of rendering processes, facilitating faster and more effective training for the student. This strategy requires the teacher and student to share a similar U-Net backbone.
- Auxiliary Unlabeled Dataset: To further enhance the student's learning without the cost of generating high-quality ground truth for every sample, an auxiliary dataset of unlabeled noisy images is used. The teacher model generates noise-free outputs for these images, allowing the student to learn from the teacher's denoising capabilities on a larger scale.
The teacher and student models themselves are based on kernel prediction networks, using a pyramid architecture for denoising. The student model is derived from a single layer of the teacher's kernel predictor (kernel size 5) with reduced channel counts in its embedding and U-Net components to ensure real-time performance.
Datasets
- Labeled Training Dataset: Generated using 6 publicly available Tungsten scenes.
- Auxiliary Unlabeled Dataset: Generated using 4 publicly available Tungsten scenes and 4 publicly available PBRT scenes.
- Rendering Details: For both datasets, scenes were rendered with 6000 samples per pixel (spp) using the Falcor renderer, providing input noise, G-Buffer information (albedo, normal, depth), and reference frames. Training data consists of 600 10-frame sequences, with randomization of materials and textures for diversity.
- Evaluation: The models were evaluated on test scenes including Bistro, Zero-Day, Classroom, and Dining room.
Key Insights and Ideas
- Knowledge Distillation for Real-Time Graphics: The paper demonstrates that knowledge distillation is a powerful technique not just for model compression in general, but specifically for enabling high-quality real-time rendering applications.
- Render-Awareness is Crucial: Simply compressing a denoiser isn't enough. The "render-aware" aspect, through initialization and the local-to-global distillation, is key to preserving rendering-specific details and lighting effects.
- GANs for Global Image Quality: Using GANs for global distillation helps the student model learn to produce images that are perceptually similar to the teacher's, not just locally accurate. This is vital for avoiding artifacts and achieving overall visual coherence.
- Auxiliary Data for Efficiency: Leveraging unlabeled data with a teacher model is an effective way to expand the training set and improve generalization without the prohibitive cost of generating high-spp ground truth for every frame.
Key Results
- Real-Time Performance: The student model, optimized with TensorRT, achieves a frame rate of approximately 10 ms per full HD frame (1920x1080), comparable to NVIDIA's OptiX denoiser.
- State-of-the-Art Quality: The RAKD framework achieves state-of-the-art denoising quality, outperforming other real-time methods in PSNR and SSIM metrics (as shown in Table 1).
- Detail Preservation: The method excels at preserving high-frequency details and complex lighting/shading effects, outperforming methods like ONND and WS, and showing better detail retention than NPPD, which can over-blur.
- Ablation Studies:
- Removing GAN regularization (w/o D) leads to overfitting and poorer generalization.
- Removing global loss (w/o G) results in blurry edges and less effective detail learning.
- Using Xavier initialization (w/o I) leads to suboptimal convergence.
Limitations and Future Work
- Specular Reflections: The current method struggles with scenes featuring strong specular reflections (e.g., mirrors), potentially due to relying solely on primary ray intersections. The authors suggest splitting diffuse and specular radiance as a future direction.
- Temporal Stability: While temporal loss is incorporated, the 10-frame sequences used for training limit temporal stability compared to methods using longer sequences (e.g., 64 frames). Future work could explore longer temporal contexts.
- Generalizability: The authors note that the teacher's denoising performance is critical; if the teacher struggles, the auxiliary dataset might not be as beneficial. Future work could explore ways to mitigate this dependency.
- Backbone Independence: While the current RAKD leverages a shared U-Net backbone for render-aware initialization, exploring ways to apply this framework to different denoising backbones would be valuable.
Summary updated