TL;DR
The Context-based NeRF Compression (CNC) framework achieves a staggering 100× size reduction for Instant-NGP models while maintaining or even improving image fidelity. By treating NeRF feature embeddings as data to be entropy-encoded and leveraging spatial dependencies across resolution levels and dimensions, the authors significantly reduce the storage footprint of high-quality 3D scene representations.
Introduction: The Speed vs. Storage Dilemma
Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, but they face a persistent "speed vs. storage" trade-off. While original NeRFs are compact (MLP-only) but slow to render, modern "hybrid" approaches like Instant-NGP (INGP) achieve real-time speeds by using explicit multi-resolution hash tables (feature embeddings).
However, these embeddings are massive, often consuming hundreds of megabytes per scene. This storage bloat hinders deployment on mobile devices and web streaming. While previous methods attempted pruning or binarization, they often overlooked the inherent spatial and structural redundancies within the grid. The CNC framework addresses this by asking: How much of this data can be predicted from its neighbors?
The CNC Framework: Context-Based Compression
The CNC framework introduces a sophisticated pipeline for entropy minimization. Instead of simple quantization, it builds Context Models to predict the probability of each bit, allowing for highly efficient arithmetic coding.
Binarization & Entropy Estimation
Feature embeddings are binarized to $\{-1, +1\}$ and modeled using a Bernoulli distribution. A differentiable "Bit Estimator" calculates entropy during training, enabling the model to optimize for both image quality and storage size simultaneously.
Hierarchical Context Models
To capture redundancies, CNC employs two primary context strategies:
- Level-wise Context Models: Since INGP uses a coarse-to-fine grid, features at finer levels are often related to their "parents" at coarser levels. A "Context Fuser" MLP predicts the probability of fine-grained vertices based on these coarser dependencies.
- Dimension-wise Context Models: To capture cross-dimensional dependencies, the model projects 3D voxel features onto 2D planes (Projected Voxel Features). These 2D maps act as context to help compress 2D tri-plane features.
Hash Fusion with Occupancy Grids
CNC repurposes the existing occupancy grid—already used in INGP for fast rendering—as a spatial prior. If a vertex falls in empty space, the model assigns it a weight of zero, effectively focusing the "bit budget" on actual object surfaces and mitigating the impact of hash collisions.
Key Results
The authors evaluated CNC on the Synthetic-NeRF and Tanks and Temples benchmarks, setting a new state-of-the-art (SOTA) for NeRF compression:
- Massive Shrinkage: Achieved 100× size reduction on Synthetic-NeRF and 70× on Tanks and Temples compared to the original Instant-NGP.
- Superior Efficiency: CNC reduced storage size by over 80% compared to BiRF (the previous leading method) while maintaining high visual fidelity.
- Compression as Regularization: Interestingly, compressed models sometimes outperformed uncompressed baselines in PSNR. The entropy constraint appears to act as a form of regularization, preventing over-fitting to noise in training views.
Limitations and Future Work
While CNC represents a major leap forward, it introduces some overhead. Training is approximately 1.3× slower than the baseline due to the context models, and there is a brief (~1 second) decoding latency before rendering can begin. Future research may focus on jointly optimizing context models and rendering MLPs to further reduce training time and latency.
Conclusion
CNC demonstrates that NeRF feature embeddings are highly redundant and can be compressed far beyond simple quantization. By leveraging spatial and dimensional context, high-quality 3D scenes can now be represented in less than 1MB, paving the way for efficient mobile and web-based 3D applications.
Summary updated