Abstract
To enrich the functionalities of traditional cameras, light field cameras record both the intensity and direction of light rays, so that images can be rendered with user-defined camera parameters via computations. The added capability and flexibility are gained at the cost of gathering typically more than $100\times$ greater amount of information than conventional images. To cope with this issue, several light field compression schemes have been introduced. However, their ways of exploiting correlations of multidimensional light field data are complex and are hence not suited for inexpensive light field cameras. In this work, we propose a novel $\ell_\infty$-constrained light-field image compression system that has a very low-complexity DPCM encoder and a CNN-based deep decoder. Targeting high-fidelity reconstruction, the CNN decoder capitalizes on the $\ell_\infty$-constraint and light field properties to remove the compression artifacts and achieves significantly better performance than existing state-of-the-art $\ell_2$-based light field compression methods.
Introduction
Light field (LF) cameras represent a significant evolution in photography, capturing the direction of every light ray in addition to intensity. This four-dimensional (4D) data allows for post-processing capabilities such as refocusing and viewpoint shifting. However, the massive volume of LF data—often 100x larger than standard photos—presents a major compression challenge. Existing high-performance codecs like HEVC are computationally expensive at the encoder side, making them impractical for consumer-grade cameras. This research introduces an asymmetric system that pairs a low-complexity encoder with a sophisticated deep-learning decoder to achieve high-fidelity reconstruction.
System Architecture
The proposed system, $\ell^{4d}_\infty$DNet, shifts the computational burden from the camera to the playback device. It consists of a near-lossless encoder and a deep-learning-based decoder designed to exploit the unique geometry of light fields.
$\ell_\infty$-constrained Encoding
The encoder utilizes a simple predictive coding method (DPCM) that guesses pixel values based on neighbors and saves only the difference. By employing an $\ell_\infty$ constraint, the system guarantees that no reconstructed pixel deviates from the original by more than a user-defined threshold ($\tau$). This mathematical boundary acts as a "safety net" for the deep decoder, preventing the generation of unrealistic artifacts or "hallucinations" common in purely generative models.
Deep Decoding with Mixture of Experts
To handle the 4D nature of light fields efficiently, the decoder avoids expensive 4D convolutions by decomposing the data into four 2D representations: Sub-Aperture Images (SAI), horizontal and vertical Epipolar Plane Images (EPI-H, EPI-V), and Micro Images (MI).
The decoder employs a "Mixture of Experts" approach, where four dedicated ExpertNets process these representations to mitigate compression artifacts. A GateNet then determines the optimal pixel-wise weighting for these experts based on the local scene structure. A final Truncated Activation layer ensures the output strictly adheres to the $\ell_\infty$ error bounds established during encoding.
Experimental Results
The system was evaluated using the EPFL light field dataset and benchmarks from the ICME 2016 light-field compression challenge.
- Superior Quality: The $\ell^{4d}_\infty$DNet outperformed the HEVC gold standard by an average of 3.21 dB in BD-PSNR.
- Encoding Speed: The DPCM encoder is significantly faster than HEVC, reducing encoding time from up to 60 minutes to just 30-60 seconds per light field.
- Efficiency: The model is remarkably compact, utilizing only 1.02M parameters, which is roughly half the size of competing deep-learning models like SDNet.
Conclusion
By combining a "dumb," ultra-fast encoder with a "smart" geometry-aware decoder, this research provides a viable solution for high-fidelity light field compression on resource-constrained devices. The integration of mathematical $\ell_\infty$ constraints with deep learning ensures both high performance and data integrity. Future work will extend this framework to full-color implementations and further optimize decoder speed for mobile platforms.
Summary updated