GAN-based multi-view video coding with spatio-temporal EPI reconstruction

Chengdong Lan, Hao Yan, Cheng Luo, Tiesong Zhao

TL;DR

This paper introduces a novel multi-view video coding (MVC) method that leverages Generative Adversarial Networks (GANs) to reconstruct intermediate views with higher accuracy and reduced bitrate. By constructing spatio-temporal Epipolar Plane Images (EPIs) and extracting latent codes from them, the method effectively captures inter-view correlations, leading to significant improvements in Rate-Distortion (RD) performance compared to existing state-of-the-art techniques.

What Problem Does This Paper Address?

Capturing video from multiple viewpoints (multi-view video) offers a more immersive experience but drastically increases data storage and transmission requirements. To mitigate this, researchers often skip intermediate views during compression and reconstruct them at the decoder using "Side Information" (SI). Traditionally, depth maps are used for SI, but their reconstruction is prone to inaccuracies, and the resulting compressed depth maps can still be high in bitrate. Existing methods that use low-resolution images as SI also face bitrate overhead. This paper addresses the challenge of achieving both high reconstruction accuracy and low bitrates for intermediate views in MVC, particularly by improving the quality and efficiency of Side Information.

Methods

The proposed method, named MVLL (Multi-View Latent code Learning), introduces several key innovations:

  • Spatio-Temporal EPI Construction: Instead of traditional EPIs, the authors construct spatio-temporal EPIs by grouping image strips from different views and stacking successive spatial EPIs. This approach aims to better capture both spatial and temporal correlations across views.
  • GAN-based Latent Code Extraction and Compression: The spatio-temporal EPI is fed into a convolutional network to extract a latent code. This latent code, representing high-level semantic features, is then compressed using an auto-encoder. The core idea is to use this compressed latent code as the Side Information.
  • GAN-based EPI Reconstruction: At the decoder, a GAN generator reconstructs the intermediate view's EPI from the compressed latent code. A discriminator network is employed to ensure the reconstructed EPI is as close as possible to the original.
  • Joint Training and RD Optimization: The entire framework, including the latent code extraction/compression and GAN-based reconstruction, is trained end-to-end. A joint objective function is used to optimize for both reconstruction quality (distortion loss) and bitrate (entropy loss), achieving an optimal trade-off.

Datasets

The paper uses five typical multi-view video sequences for training and evaluation: Scene_Door_Flowers, Scene_Leaving_Laptop, Scene_Outdoor, Champagne_Tower, and Dog. These sequences are processed into EPI images for training and testing. The authors do not introduce new benchmarks but evaluate against existing standards and methods.

Key Insights and Ideas

  • Latent Codes as Effective Side Information: The paper demonstrates that compressed latent codes extracted from spatio-temporal EPIs can serve as highly effective Side Information for reconstructing intermediate views, outperforming traditional depth-map-based methods.
  • Spatio-Temporal EPIs Capture Richer Correlations: By combining spatial and temporal information within the EPI structure, the method gains a more comprehensive understanding of the scene, leading to better feature extraction.
  • GANs for High-Fidelity Reconstruction: The use of GANs for EPI reconstruction allows for generating realistic and detailed intermediate views, leveraging the GAN's ability to learn complex data distributions and generate high-quality images.
  • End-to-End RD Optimization is Crucial: Jointly training the encoder (latent code extraction/compression) and decoder (GAN reconstruction) with an explicit rate-distortion objective function is key to achieving the best balance between compression efficiency and visual quality.

Key Results

The proposed MVLL method demonstrates significant improvements in Rate-Distortion (RD) performance compared to state-of-the-art methods like MV-HEVC, MVD, DCVC, and TSAN.

  • Bitrate Reduction: The paper reports bitrate savings of up to 44.14% at the same PSNR and 32.59% at the same SSIM compared to MVD.
  • Quality Improvement: The method also achieves notable quality increments in terms of PSNR and SSIM at equivalent bitrates.
  • Computational Efficiency: The encoder is reported to be faster than traditional hand-crafted MVC standards like MV-HEVC and 3D-HEVC.

Limitations and Future Work

  • Computational Complexity: While the encoder is faster than some traditional methods, the computational complexity of GANs can still be a concern for real-time applications. The paper mentions using a single 2080TI GPU for evaluation.
  • Ablation Study on EPI Structure: The paper includes an ablation study showing that the spatio-temporal EPI structure outperforms traditional and spatial EPIs.
  • Ablation Study on GAN vs. Autoencoder: Another ablation study confirms that the GAN-based reconstruction with a discriminator network yields better RD performance than a pure autoencoder approach.
  • Future Work: The authors express hope that their work provides an innovative methodology for deep-learning-based multi-view video coding. Further research could explore more efficient GAN architectures, adaptive latent code compression strategies, and integration with other advanced video coding techniques.

Summary updated

BibTeX
@inproceedings{lan2024gan,
  author = {Chengdong Lan and Hao Yan and Cheng Luo and Tiesong Zhao},
  title = {{GAN-based multi-view video coding with spatio-temporal EPI reconstruction}},
  booktitle = {Signal},
  year = {2024},
  doi = {10.1016/j.image.2024.117242},
}
Figure