Abstract
This paper presents a novel multi-view video coding (MVC) framework, Multi-View Latent Learning (MVLL), which leverages Generative Adversarial Networks (GANs) to reconstruct skipped intermediate viewpoints. By transmitting compact GAN latent codes derived from spatio-temporal Epipolar Plane Images (EPIs) as side information, the proposed method achieves superior reconstruction quality at significantly lower bitrates compared to traditional depth-based or deep-learning approaches.
Introduction
Multi-view video, which captures scenes from multiple simultaneous angles, is a cornerstone for immersive technologies like Virtual Reality (VR) and free-viewpoint television. However, the sheer volume of data presents a massive storage and transmission challenge. Traditional solutions often involve skipping intermediate viewpoints at the encoder and attempting to reconstruct them at the decoder using Side Information (SI).
The industry-standard Multi-view plus Depth (MVD) approach uses depth maps as SI. Unfortunately, calculating accurate depth is computationally intensive and prone to errors that cause visual artifacts. Alternative methods using low-resolution images often suffer from high bitrate overhead. The challenge lies in creating a representation of missing information that is both extremely compact and highly descriptive.
Proposed Methodology
The MVLL framework shifts the reconstruction paradigm from geometric depth-mapping to generative modeling. The process is divided into three primary stages: construction, extraction, and reconstruction.
1. Spatio-Temporal EPI Construction
Instead of raw video frames, the system utilizes Epipolar Plane Images (EPIs). By grouping pixel rows from different viewpoints, EPIs transform spatial correlations between views into linear structures. These are stacked over time to capture temporal correlations, creating a rich spatio-temporal representation of the scene.
2. Latent Code Extraction and Compression
A Convolutional Neural Network (CNN) encoder extracts a high-level "latent code" ($Z$) from the spatio-temporal EPI. This code serves as a compact mathematical summary of the scene's structure. To ensure efficient transmission, this code is further compressed via an auto-encoder and quantized.
3. GAN-Based Reconstruction
At the decoder, a GAN generator receives the compressed latent code along with adjacent reference views. It uses this information to synthesize the missing intermediate view. The system is trained end-to-end using a joint Rate-Distortion (RD) loss function that balances the entropy of the latent code with the visual fidelity (MSE and VGG feature loss) of the output. An adversarial discriminator ensures the reconstructed views remain sharp and realistic.
Experimental Results
The MVLL method was evaluated against state-of-the-art codecs, including MV-HEVC and MVD, using standard multi-view datasets and Common Test Conditions (CTC).
- Bitrate Efficiency: MVLL achieved average bitrate savings (BDBR) of 20.44% over MV-HEVC and 44.14% over depth-based MVD for equivalent image quality (PSNR).
- Perceptual Quality: When evaluated using SSIM, which more closely tracks human visual perception, the method saved 32.59% in bitrate compared to MVD.
- Computational Performance: Reconstruction takes approximately 0.204 seconds per frame on a single NVIDIA 2080TI GPU, outperforming several traditional hand-crafted MVC standards in speed.
Ablation studies confirmed that the spatio-temporal EPI structure is critical to these gains, significantly outperforming models that relied solely on spatial data or traditional image structures.
Conclusion
By replacing explicit geometric modeling with generative latent learning, MVLL offers a more robust and efficient path for multi-view video compression. While current implementation requires modern GPU hardware for decoding, the significant gains in bitrate and visual quality suggest that latent learning is a promising direction for future immersive video standards.
Summary updated