3D Snapshot: Invertible Embedding of 3D Neural Representations in a Single Image

Yuqin Lu, Bailin Deng, Zhixuan Zhong, Tianle Zhang, Yuhui Quan, Hongmin Cai, Shengfeng HeIEEE TPAMI 2024 · DOI · OpenReview

Abstract

3D Snapshot introduces a novel framework for embedding complex 3D neural representations, such as Neural Radiance Fields (NeRFs), into a single, seemingly ordinary 2D "host" image. By utilizing invertible neural networks (INNs), the method hides 3D scene data as imperceptible high-frequency patterns. This allows a standard photo to serve as a container for a full 3D scene, enabling high-quality arbitrary view synthesis and real-time decoding on mobile devices without the storage overhead of traditional 3D formats.

Introduction

Neural Radiance Fields (NeRFs) have transformed 3D scene reconstruction but often suffer from large storage requirements and specialized file formats. While video-based invertible methods exist for hiding visual data, they typically face a "capacity wall" where restoration quality degrades significantly as more views are added. 3D Snapshot addresses this challenge by embedding the continuous neural representation itself into a single 2D image, bypassing the limitations of discrete frame storage and allowing for arbitrary view synthesis from a single file.

Methodology

The core of 3D Snapshot is an Invertible Neural Network (INN) architecture comprising 16 invertible blocks. These blocks utilize coupling layers—a standard technique in flow-based generative models—to ensure the transformation between the 3D representation and the 2D host image is mathematically reversible.

Embedding and Restoration of 3D scenes (Chair, Drums, Ship) using 3D Snapshot.

Within each block, 5-layer densely connected convolutional networks provide the non-linearity required to handle complex visual data. The 3D information is encoded as high-frequency, imperceptible patterns within the host image. Optimization is performed using the Adam optimizer with a dynamic "target constant noise tensor" to stabilize the early stages of training.

Experiments and Results

The framework was evaluated on the NeRF-Synthetic and Tanks and Temples datasets. A key finding is the robustness of the method regarding the "host" image; the choice of the initial 2D view does not significantly impact the final 3D reconstruction quality. For instance, in the "Hotdog" scene, the Peak Signal-to-Noise Ratio (PSNR) remained remarkably stable (between 29.97 and 30.14) regardless of the host perspective.

Qualitative comparison of 3D Snapshot against ground truth (GT) and state-of-the-art NeRF variants.

As shown in the qualitative results, 3D Snapshot achieves rendering quality comparable to state-of-the-art models like TensoRF and Plenoxels while maintaining a significantly smaller and more portable model size. It outperforms previous video-based embedding methods, particularly in maintaining visual consistency when synthesizing views that deviate significantly from the original host perspective.

Discussion and Limitations

While 3D Snapshot enables mobile deployment—demonstrated via an app that decodes printed snapshots in real-time—computational constraints on mobile hardware currently necessitate reduced rendering resolutions (e.g., 380 x 380) to maintain fluency. Additionally, while the method handles complex geometry, extremely high-entropy or massive-scale environments may eventually push the limits of what can be hidden imperceptibly in a single 2D image.

Conclusion

3D Snapshot provides a bridge between physical media and immersive digital content. By turning 2D images into "portals" for 3D scenes, this technology offers a portable, efficient, and visually seamless way to share and store 3D neural representations, potentially serving as a foundational technology for future metaverse applications.

Summary updated

Abstract

3D neural rendering enables photo-realistic reconstruction of a specific scene by encoding discontinuous inputs into a neural representation. Despite the remarkable rendering results, the storage of network parameters is not transmission-friendly and not extendable to metaverse applications. In this paper, we propose an invertible neural rendering approach that enables generating an interactive 3D model from a single image (i.e., 3D Snapshot). Our idea is to distill a pre-trained neural rendering model (e.g., NeRF) into a visualizable image form that can then be easily inverted back to a neural network. To this end, we first present a neural image distillation method to optimize three neural planes for representing the original neural rendering model. However, this representation is noisy and visually meaningless. We thus propose a dynamic invertible neural network to embed this noisy representation into a plausible image representation of the scene. We demonstrate promising reconstruction quality quantitatively and qualitatively, by comparing to the original neural rendering model, as well as video-based invertible methods. On the other hand, our method can store dozens of NeRFs with a compact restoration network (5 MB), and embedding each 3D scene takes up only 160 KB of storage. More importantly, our approach is the first solution that allows embedding a neural rendering model into image representations, which enables applications like creating an interactive 3D model from a printed image in the metaverse.
BibTeX
@article{lu2024d,
  author = {Yuqin Lu and Bailin Deng and Zhixuan Zhong and Tianle Zhang and Yuhui Quan and Hongmin Cai and Shengfeng He},
  title = {{3D Snapshot: Invertible Embedding of 3D Neural Representations in a Single Image}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3411051},
}
Figure