UHDNeRF: Ultra-High-Definition Neural Radiance Fields

Quewei Li, Feichao Li, Jie Guo, Yanwen GuoICCV 2023 · OpenReview

Abstract

We propose UHDNeRF, a new framework for novel view synthesis on the challenging ultra-high-resolution (e.g., 4K) real-world scenes. Previous NeRF methods are not specifically designed for rendering on extremely high resolutions, leading to blurry results with notable detail-losing problems even though trained on 4K images. This is mainly due to the mismatch between the high-resolution inputs and the low-dimensional volumetric representation. To address this issue, we introduce an adaptive implicit-explicit scene representation with which an explicit sparse point cloud is used to boost the performance of an implicit volume on modeling subtle details. Specifically, we reconstruct the complex real-world scene with a frequency separation strategy: the implicit volume learns to represent the low-frequency properties of the whole scene, and the sparse point cloud is used for reproducing high-frequency details. To better explore the information embedded in the point cloud, we extract a global structure feature and a local point-wise feature from the point cloud for each sample located in the high-frequency regions. Furthermore, a patch-based sampling strategy is introduced to reduce the computational cost. The high-fidelity rendering results demonstrate the superiority of our method for retaining high-frequency details at 4K ultra-high-resolution scenarios against state-of-the-art NeRF-based solutions.

Introduction: The Resolution Wall in Neural Rendering

While Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction, they encounter a significant "resolution wall" when scaling from standard 1K resolutions to 4K Ultra-High-Definition (UHD). Standard NeRF architectures require an impractical number of samples per ray and massive MLP evaluations to capture 4K details, leading to prohibitive rendering times. Conversely, hybrid methods utilizing explicit data structures like dense voxel grids or point clouds suffer from unacceptable memory overhead at these scales. As a result, most current NeRF implementations produce blurry textures when tasked with high-resolution rendering.

Methodology: Frequency Separation and Adaptive Branching

UHDNeRF addresses these limitations through an adaptive implicit-explicit scene representation governed by a frequency separation strategy.

  1. Frequency Separation: The scene is decomposed into two components. An implicit volume (MLP-based NeRF) models low-frequency properties such as smooth surfaces and global illumination. A sparse point cloud is dedicated to representing high-frequency details like sharp edges and fine textures.
  2. Adaptive Branching: During rendering, the system adaptively chooses a processing path for each sample point along a camera ray. If a sample has neighboring points in the sparse cloud, it is processed by the High-frequency branch, which integrates point-based features. Otherwise, it defaults to the Low-frequency branch (purely implicit).
  3. Feature Extraction: To maximize the utility of the sparse points, UHDNeRF extracts both a global structure feature (capturing 3D layout) and a local point-wise feature. A Conditional Feature Transfer (CFT) module ensures translation invariance, improving generalization.
  4. Patch-based Sampling: To optimize training, UHDNeRF samples 100x100 patches rather than random pixels. This reduces the computational cost of point cloud queries, as adjacent rays within a patch frequently share the same point neighbors.

Key Results and Visual Fidelity

UHDNeRF consistently outperforms state-of-the-art models, including MipNeRF-360, Instant NGP, and NeRF-SR, in both quantitative and qualitative assessments. On the LLFF dataset, UHDNeRF achieved a PSNR of 29.03, significantly higher than Instant NGP+ (27.63) and MipNeRF-360+ (24.79).

Qualitative comparison of UHDNeRF against state-of-the-art methods. The composite shows the full scene and zoomed-in regions (red and green boxes). UHDNeRF (rightmost columns) preserves sharp details in high-frequency areas like text and small objects, whereas competing methods produce blurry results.

Qualitatively, the model demonstrates a stark improvement in texture preservation. It successfully renders fine details, such as the individual petals of an orchid or the jagged edges of a stone fortress, where other models produce "mushy" artifacts. Furthermore, UHDNeRF is memory-efficient; by focusing the point cloud only on high-frequency regions, it requires approximately 13-15 GB of cache memory for 4K rendering, significantly less than dense point-based methods like Point-NeRF.

Conclusion and Future Directions

UHDNeRF demonstrates that a hybrid, frequency-aware approach is essential for achieving true 4K clarity in neural radiance fields. By using a sparse point cloud as an "anchor" for high-frequency details, the model avoids the blurring common in purely implicit representations. While the method currently depends on COLMAP for initialization and faces challenges in sparse-view 360° scenes, it sets a new benchmark for high-fidelity 3D reconstruction in ultra-high-definition scenarios.

Summary updated

Abstract

We propose UHDNeRF, a new framework for novel view synthesis on the challenging ultra-high-resolution (e.g., 4K) real-world scenes. Previous NeRF methods are not specifically designed for rendering on extremely high resolutions, leading to burry results with notable detail-losing problems even though trained on 4K images. This is mainly due to the mismatch between the high-resolution inputs and the low-dimensional volumetric representation. To address this issue, we introduce an adaptive implicit-explicit scene representation with which an explicit sparse point cloud is used to boost the performance of an implicit volume on modeling subtle details. Specifically, we reconstruct the complex real-world scene with a frequency separation strategy that the implicit volume learns to represent the low-frequency properties of the whole scene, and the sparse point cloud is used for reproducing high-frequency details. To better explore the information embedded in the point cloud, we extract a global structure feature and a local point-wise feature from the point cloud for each sample located in the high-frequency regions. Furthermore, a patch-based sampling strategy is introduced to reduce the computational cost. The high-fidelity rendering results demonstrate the superiority of our method for retaining high-frequency details at 4K ultra-high-resolution scenarios against state-of-the-art NeRF-based solutions.
BibTeX
@inproceedings{li2023uhdnerf,
  author = {Quewei Li and Feichao Li and Jie Guo and Yanwen Guo},
  title = {{UHDNeRF: Ultra-High-Definition Neural Radiance Fields}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=BNJnDrpZIS},
}
Figure