TL;DR
This paper introduces a novel framework for accurately estimating rendering errors in Monte Carlo images that have been denoised using deep learning. This allows for a robust stopping criterion that automatically terminates rendering when a user-defined quality threshold is met, eliminating the need for manual trial-and-error.
What Problem Does This Paper Address?
Modern rendering pipelines heavily rely on Monte Carlo methods (like path tracing) for realistic image synthesis. However, these methods are computationally expensive, requiring many samples per pixel to reduce noise. Deep learning-based denoisers have become essential for speeding up this process by reducing noise with fewer samples. The challenge is that denoisers introduce bias, making traditional error estimation methods (which rely on variance alone) insufficient. Without a reliable way to estimate the actual error of a denoised image, users must resort to guesswork to determine how many samples are needed to achieve a desired quality, leading to wasted render time or suboptimal results. This paper tackles the problem of providing a practical, automatic stopping criterion for denoised Monte Carlo rendering by accurately estimating the combined error (bias and variance).
Methods
The core of the paper is a new error estimation framework built upon Stein's Unbiased Risk Estimate (SURE). Here's a breakdown of their approach:
- Theoretical Foundation: They posit that the squared error of a denoised pixel, under reasonable assumptions about the input, follows a noncentral chi-squared distribution. This is a key departure from traditional methods that assume a simple chi-squared distribution based on variance alone.
- SURE for Error Estimation: They adapt SURE, a statistical tool for estimating the mean squared error of an estimator, to estimate the expected squared error of denoised pixels.
- Addressing Variance: Per-pixel SURE estimates are notoriously noisy. To combat this, they employ a hierarchical block-averaging strategy using a k-d tree. This process adaptively splits image regions based on the reliability of SURE estimates, ensuring that averaging occurs only where it's statistically sound.
- Incorporating Spatial Contrast Sensitivity: Recognizing that not all errors are equally perceptible, they augment their error metric by incorporating spatial contrast sensitivity, inspired by the ILIP error metric. This helps prioritize errors that are visually significant to human perception.
- Estimating Distribution Parameters: They estimate the parameters (scale and noncentrality) of the noncentral chi-squared distribution for each pixel. This allows them to model the entire error distribution, not just its mean.
- Stopping Criterion: Using the estimated error distribution parameters, they compute the cumulative distribution function (CDF) of the error for the entire image. The stopping criterion then checks if a user-defined percentile (e.g., 99.9%) of pixels fall below a specified error threshold.
Datasets
The paper evaluates their method on a set of 20 publicly available scenes. No new datasets were introduced.
Key Insights and Ideas
- Noncentral Chi-Squared Distribution is Key: The realization that denoised pixel errors follow a noncentral chi-squared distribution, rather than a simple chi-squared one, is crucial. This distribution inherently accounts for bias, which is introduced by denoisers.
- SURE's Potential and Pitfalls: SURE provides a theoretically sound way to estimate expected squared error, but its high variance necessitates sophisticated aggregation techniques like hierarchical block averaging.
- Perceptual Relevance Matters: Incorporating spatial contrast sensitivity into the error metric makes the stopping criterion more aligned with human perception, leading to more visually pleasing results.
- Automatic Quality Control: The framework enables automatic rendering termination based on a user-defined error threshold, removing the guesswork and saving significant artist time and computational resources.
Key Results
- The proposed stopping criterion shows close agreement with the ground truth in 87% of cases for the SURE-based method and 82% for the SUREM (perceptually enhanced) method, meaning their terminal sample counts are within one-third or one step (32 spp) of the true optimal count.
- The framework incurs a small runtime overhead (around 4% for a representative scene) compared to the rendering process itself, and a minimal overhead (1-2%) when integrated with adaptive sampling methods.
- Qualitative comparisons demonstrate that their method accurately estimates error percentiles and that ignoring bias (as in previous variance-only methods) leads to underestimation of error.
Limitations and Future Work
- SURE Assumption: The method relies on the assumption that sample means are normally distributed. While this holds for many samples, it can be problematic for low sample counts or distributions with infinite variance.
- Splitting Criterion Failure: The block-splitting criterion can fail if SURE estimates are erroneously small or negative, which can happen due to bias or variance in the estimates.
- Linearity Constraint: The incorporation of spatial contrast sensitivity was limited to linear operations. Future work could explore more sophisticated perceptual metrics that might involve non-linear operations.
- RGB Color Space: The current implementation uses the linear RGB color space for error estimation. Exploring other color spaces for improved perceptual relevance is a potential future direction.
Summary updated