Abstract
We propose a novel approach for embedding novel views in a single JPEG image while preserving the perceptual fidelity of the modified JPEG image and the restored novel views. We adopt the popular novel view synthesis representation of multiplane images (MPIs). Our model first encodes 32 MPI layers (totally 128 channels) into a 3-channel JPEG image that can be decoded for MPIs to render novel views, with an embedding capacity of 1024 bits per pixel. We conducted experiments on public datasets with different novel view synthesis methods, and the results show that the proposed method can restore high-fidelity novel views from a slightly modified JPEG image. Furthermore, our method is robust to JPEG compression, color adjusting, and cropping.
Introduction
While novel view synthesis (NVS) has made massive strides, sharing 3D content remains a friction-filled process. Most social media platforms and web browsers are built for 2D formats like JPEG. Sharing a 3D photo typically requires sharing a video of a camera swing or a proprietary file format that lacks broad compatibility.
The challenge is one of capacity and compatibility. A Multiplane Image (MPI)—a popular 3D representation—consists of multiple layers of RGB and Alpha (transparency) data. For a 32-layer MPI, that is 128 channels of data. Compressing those 128 channels into a 3-channel JPEG without creating visual artifacts or losing 3D information is an extreme steganography problem, requiring an embedding capacity of roughly 1024 bits per pixel.
Methodology
Researchers from HKUST proposed an end-to-end deep learning framework consisting of three main components:
- Embedding Network (Encoder): Fuses the 32-layer MPI and a reference 2D image using a feature fusion mechanism that weights RGB layers by their alpha transparency, prioritizing visible scene elements.
- Restoration Network (Decoder): A lightweight network (6.3M parameters) designed for real-time extraction of the MPI from the modified JPEG on mobile devices.
- Discriminator: Ensures the embedded JPEG remains perceptually indistinguishable from the original.
To handle the lossy nature of JPEG, the team utilized a differentiable JPEG approximation during training. They also introduced a Frequency Domain Loss to prevent high-frequency noise and a Render Loss to ensure the most important 3D features are preserved for viewing.
Results and Evaluation
The model was trained on the RealEstate10K dataset and tested for generalization on the LLFF dataset. The system achieves a PSNR of 34.6 dB for the embedding image and 36.7 dB for the restored novel views, indicating high fidelity in both the host image and the 3D reconstruction.
The restoration network is highly efficient, with an inference time of only 0.003 seconds. This efficiency makes real-time 3D viewing on smartphones feasible. Compared to alternatives like Layered Depth Images (LDI), this approach has a significantly smaller model footprint (6.3M vs 438M parameters).
Conclusion
This approach successfully hides complex 3D data within standard JPEGs, surviving common social media "attacks" like compression and cropping. While limited by the inherent constraints of MPIs (e.g., fixed depth range) and potential degradation under extreme compression, it offers a low-barrier path to sharing 3D content across existing 2D-centric platforms.
Summary updated