How Far can we Compress Instant-NGP-Based NeRF?

Yihang Chen, Qianyi Wu, Mehrtash Harandi, Jianfei Cai

TL;DR

The Context-based NeRF Compression (CNC) framework achieves a staggering 100× size reduction for Instant-NGP models while maintaining or even improving image fidelity. By treating NeRF feature embeddings as data to be entropy-encoded and leveraging spatial dependencies across resolution levels and dimensions, the authors significantly reduce the storage footprint of high-quality 3D scene representations.

Introduction: The Speed vs. Storage Dilemma

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, but they face a persistent "speed vs. storage" trade-off. While original NeRFs are compact (MLP-only) but slow to render, modern "hybrid" approaches like Instant-NGP (INGP) achieve real-time speeds by using explicit multi-resolution hash tables (feature embeddings).

However, these embeddings are massive, often consuming hundreds of megabytes per scene. This storage bloat hinders deployment on mobile devices and web streaming. While previous methods attempted pruning or binarization, they often overlooked the inherent spatial and structural redundancies within the grid. The CNC framework addresses this by asking: How much of this data can be predicted from its neighbors?

Overview of the CNC framework and its compression performance on standard benchmarks.

The CNC Framework: Context-Based Compression

The CNC framework introduces a sophisticated pipeline for entropy minimization. Instead of simple quantization, it builds Context Models to predict the probability of each bit, allowing for highly efficient arithmetic coding.

Binarization & Entropy Estimation

Feature embeddings are binarized to $\{-1, +1\}$ and modeled using a Bernoulli distribution. A differentiable "Bit Estimator" calculates entropy during training, enabling the model to optimize for both image quality and storage size simultaneously.

Hierarchical Context Models

To capture redundancies, CNC employs two primary context strategies:

  1. Level-wise Context Models: Since INGP uses a coarse-to-fine grid, features at finer levels are often related to their "parents" at coarser levels. A "Context Fuser" MLP predicts the probability of fine-grained vertices based on these coarser dependencies.
  2. Dimension-wise Context Models: To capture cross-dimensional dependencies, the model projects 3D voxel features onto 2D planes (Projected Voxel Features). These 2D maps act as context to help compress 2D tri-plane features.
Detailed architecture of the level-wise and dimension-wise context models used for entropy minimization.

Hash Fusion with Occupancy Grids

CNC repurposes the existing occupancy grid—already used in INGP for fast rendering—as a spatial prior. If a vertex falls in empty space, the model assigns it a weight of zero, effectively focusing the "bit budget" on actual object surfaces and mitigating the impact of hash collisions.

Key Results

The authors evaluated CNC on the Synthetic-NeRF and Tanks and Temples benchmarks, setting a new state-of-the-art (SOTA) for NeRF compression:

  • Massive Shrinkage: Achieved 100× size reduction on Synthetic-NeRF and 70× on Tanks and Temples compared to the original Instant-NGP.
  • Superior Efficiency: CNC reduced storage size by over 80% compared to BiRF (the previous leading method) while maintaining high visual fidelity.
  • Compression as Regularization: Interestingly, compressed models sometimes outperformed uncompressed baselines in PSNR. The entropy constraint appears to act as a form of regularization, preventing over-fitting to noise in training views.

Limitations and Future Work

While CNC represents a major leap forward, it introduces some overhead. Training is approximately 1.3× slower than the baseline due to the context models, and there is a brief (~1 second) decoding latency before rendering can begin. Future research may focus on jointly optimizing context models and rendering MLPs to further reduce training time and latency.

Conclusion

CNC demonstrates that NeRF feature embeddings are highly redundant and can be compressed far beyond simple quantization. By leveraging spatial and dimensional context, high-quality 3D scenes can now be represented in less than 1MB, paving the way for efficient mobile and web-based 3D applications.

Summary updated

Abstract

In recent years, Neural Radiance Field (NeRF) has demonstrated remarkable capabilities in representing 3D scenes. To expedite the rendering process, learnable explicit representations have been introduced for combination with implicit NeRF representation, which however results in a large storage space requirement. In this paper, we introduce the Context-based NeRF Compression (CNC) framework, which leverages highly efficient context models to provide a storage-friendly NeRF representation. Specifically, we excavate both level-wise and dimension-wise context dependencies to enable probability prediction for information entropy reduction. Additionally, we exploit hash collision and occupancy grids as strong prior knowledge for better context modeling. To the best of our knowledge, we are the first to construct and exploit context models for NeRF compression. We achieve a size reduction of 100x and 70× with improved fidelity against the baseline Instant-NGP on Synthesic-NeRF and Tanks and Temples datasets, respectively. Additionally, we attain 86.7% and 82.3% storage size reduction against the SOTA NeRF compression method BiRF. Our code is available here: https://github.com/YihangChen-ee/CNC.
BibTeX
@inproceedings{chen2024how,
  author = {Yihang Chen and Qianyi Wu and Mehrtash Harandi and Jianfei Cai},
  title = {{How Far can we Compress Instant-NGP-Based NeRF?}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure