Mononizing binocular videos

Wenbo Hu, Menghan Xia, Chi-Wing Fu, Tien-Tsin Wong

TL;DR

This paper introduces "mono-nization," a novel technique to convert binocular (stereo) videos into regular monocular videos by imperceptibly encoding stereo information. This allows for backward-compatible storage and distribution on existing platforms, with the ability to restore the original 3D experience when a stereoscopic display is available.

What Problem Does This Paper Address?

The increasing popularity of multi-camera smartphones presents a challenge for existing single-view platforms. Migrating to multi-view content often requires significant redesign and can render older hardware obsolete. The authors propose a solution that bridges this gap by enabling binocular videos to be treated as ordinary monocular videos, ensuring backward compatibility with current infrastructure, much like the transition from black-and-white to color television. The core problem is how to embed the stereo information from a binocular video into a monocular representation in a way that is visually indistinguishable from a standard monocular video, yet allows for perfect reconstruction of the original 3D content.

Methods

The paper proposes an end-to-end, self-supervised encoding-decoding framework.

  • Encoding-Decoding Network: A deep neural network architecture is employed, comprising an encoder to "mono-nize" the binocular video and a decoder to restore the original binocular video. This approach is trained using a self-supervised learning strategy, eliminating the need for manually labeled data.
  • Pyramidal Deformable Fusion (PDF) Module: To handle the significant disparities between left and right views in binocular videos, which can challenge standard CNNs, the authors introduce the PDF module. This module leverages deformable convolutions to effectively exploit long-range correspondences between the views at multiple feature levels.
  • Quantization Layer: To mitigate artifacts introduced by the quantization process when converting floating-point network outputs to 8-bit pixel values for the mononized image, a quantization layer is incorporated. This layer is designed to be differentiable for training purposes.
  • Compression Noise Simulation (CNS) Module: To ensure that the mononized videos remain robust when compressed by lossy video codecs (like H.264, H.265, VP9, AV1), a CNS module is integrated during training. This module simulates intra-frame and inter-frame noise, making the network more resilient to codec distortions.
  • Loss Function: The framework is trained using a combination of three loss terms:
  • Monocular Loss: Ensures the mononized video visually resembles the input left view.
  • Invertibility Loss: Guarantees that the restored binocular video closely matches the original.
  • Temporal Loss: Maintains frame-to-frame coherence in both the mononized and restored videos, crucial for video sequences.

Datasets

  • Training: A custom 3D movie dataset was compiled, consisting of 122 sequences (720p resolution, 5,876 frames total) collected from Inria and YouTube. The authors filtered out sequences with unnatural disparities. Optical flows were pre-estimated using PWC-Net.
  • Image-based comparison: The Flickr1024 dataset was used for evaluating the image version of their method against other approaches.

Key Insights and Ideas

  • Implicit Stereo Encoding: The core idea is to encode stereo information not as explicit depth maps or separate views, but as subtle, nearly imperceptible visual cues within a single monocular frame. This is the key to backward compatibility.
  • Deformable Convolutions for Stereo: The Pyramidal Deformable Fusion module is a critical innovation, enabling the network to effectively learn and utilize long-range correspondences between stereo views, which is essential for handling large disparities.
  • Self-Supervised Training for Reversibility: Training the encoder and decoder jointly with an invertibility loss is a clever way to ensure that the stereo information encoded can be perfectly recovered, without requiring ground truth stereo pairs for every single frame during training.
  • Robustness to Compression: The inclusion of a Compression Noise Simulation module is vital for practical deployment, as it ensures that the mononized videos can withstand the distortions introduced by common video codecs.

Key Results

  • Mononized View Quality: The proposed method produces mononized videos that are visually indistinguishable from regular monocular videos, with significantly higher PSNR and SSIM scores compared to baseline methods like DeepSteno and InvertGray.
  • Restored View Quality: The restored binocular videos achieve high PSNR and SSIM values, demonstrating the effectiveness of the encoding and decoding process.
  • Temporal Coherence: The temporal loss and recurrent connections significantly improve temporal coherence in the generated videos, making them smoother and more natural than those produced without these components.
  • Codec Compatibility: The mononized videos are compatible with standard video codecs (H.264, H.265, VP9, AV1). At low bitrates, the method outperforms traditional side-by-side encoding and multi-view extensions (3D-MVC, MV-HEVC) in terms of restored video quality.
  • User Study: Perceptual studies confirmed that the mononized videos are indistinguishable from ground truth monocular videos, and the restored binocular videos are perceptually equivalent to the original binocular videos, with no significant difference in frame quality, temporal smoothness, or depth perception.

Limitations and Future Work

  • Manipulation Impact: Manipulating the mononized video (e.g., editing chrominance or color) can negatively impact the restorability of the binocular views, as it can disrupt the implicitly encoded stereo information.
  • Baseline Distance: While the method performs well, performance slightly degrades with increasing baseline distances, particularly for objects close to the camera. At very large baselines (e.g., 40 cm), artifacts may become noticeable.
  • Future Work: The authors plan to extend their framework to handle general multi-view videos from modern multi-camera phones. They are also interested in exploring "editable" mononized images/videos, where manipulations on the mononized view can be coherently transferred to all restored multi-views.

Summary updated

Abstract

This paper presents the idea of mono-nizing binocular videos and a framework to effectively realize it. Mono-nize means we purposely convert a binocular video into a regular monocular video with the stereo information implicitly encoded in a visual but nearly-imperceptible form. Hence, we can impartially distribute and show the mononized video as an ordinary monocular video. Unlike ordinary monocular videos, we can restore from it the original binocular video and show it on a stereoscopic display. To start, we formulate an encoding-and-decoding framework with the pyramidal deformable fusion module to exploit long-range correspondences between the left and right views, a quantization layer to suppress the restoring artifacts, and the compression noise simulation module to resist the compression noise introduced by modern video codecs. Our framework is self-supervised, as we articulate our objective function with loss terms defined on the input: a monocular term for creating the mononized video, an invertibility term for restoring the original video, and a temporal term for frame-to-frame coherence. Further, we conducted extensive experiments to evaluate our generated mononized videos and restored binocular videos for diverse types of images and 3D movies. Quantitative results on both standard metrics and user perception studies show the effectiveness of our method.
BibTeX
@article{hu2020mononizing,
  author = {Wenbo Hu and Menghan Xia and Chi-Wing Fu and Tien-Tsin Wong},
  title = {{Mononizing binocular videos}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  url = {https://openreview.net/forum?id=brl7DImAyY},
}

Loading related content...

Figure