Learning Compressible 360deg Video Isomers

Yu-Chuan Su, Kristen GraumanWorkshops @ CVPR 2018 · OpenReview

TL;DR

This paper tackles the challenge of efficiently compressing 360° videos by exploiting the fact that different rotational orientations of the same video can have vastly different compression rates. They propose a novel deep learning approach that predicts the optimal orientation for cubemap projection, leading to significant file size reductions without altering standard video codecs.

What Problem Does This Paper Address?

Standard video compression techniques, designed for traditional rectangular videos, are applied to 360° videos by first projecting the spherical content onto a flat, rectangular format. A popular method is to project the 360° sphere onto a cube and then "unwrap" this cube into a planar image (a cubemap). However, the authors observe a critical issue: the same 360° video, when rotated, can result in significantly different compression efficiencies when projected into a cubemap. This is because the discontinuities and content distribution across the boundaries of the unwrapped cubemap can drastically affect how well compression algorithms work. The problem is to find the "sweet spot" orientation that maximizes compression without requiring computationally expensive brute-force testing of all possible rotations.

Methods

The core of the paper's contribution is a learning-based approach to predict the optimal cubemap orientation. Instead of trying every possible rotation (which is computationally prohibitive), they train a Convolutional Neural Network (CNN). This CNN takes as input features extracted from the raw video clip:

  • Appearance Features: These are derived from segmentation contours, which highlight edges and object boundaries that are crucial for compression.
  • Motion Features: These are directly extracted from the motion vectors present in the input video stream's encoding. This is an efficient way to capture how pixels are moving, which is a key factor in video compression.

The video is first divided into temporal segments. For each segment, appearance and motion features are extracted and then fed into a modified VGG CNN architecture. Skip connections are used to combine low-level and high-level features. The network is trained to predict the video size for each of the 361 possible orientations, and the orientation yielding the minimum predicted size is chosen. This approach is designed to be computationally efficient, predicting the optimal orientation in less than a second.

Datasets

The authors created a dataset for their experiments by collecting 80 360° videos from YouTube, totaling 4.2 hours of 4K quality content. These videos cover a diverse range of scenarios, including aerial, underwater, sports, and news footage. The videos were initially in equirectangular format and then transcoded into cubemap format for analysis. For evaluation, they used 361 different orientations for each 2-second clip.

Key Insights and Ideas

  • "Isomers" have different compressibility: The most fundamental insight is that 360° videos, while visually equivalent under rotation (they are "isomers"), are not equivalent in terms of how well they compress. This is a direct consequence of how projection and compression algorithms interact with visual content.
  • Cubemap boundaries are compression bottlenecks: Discontinuities and abrupt motions across the seams of an unwrapped cubemap can introduce high-frequency signals that significantly harm compression efficiency.
  • Content-dependent optimization is key: The optimal orientation is not universal; it depends heavily on the specific visual content of the video clip. This makes rule-based or heuristic approaches less effective and highlights the need for a learning-based solution.
  • Motion vectors are valuable predictors: Directly using motion vectors from the encoder provides a strong signal about compressibility, as they directly inform the encoder's decisions.

Key Results

  • Significant Size Reduction: The proposed method achieves an average video size reduction of 8.43% for H264, 8.11% for HEVC, and 7.83% for VP9. This translates to substantial savings, especially for large video databases.
  • High Prediction Accuracy: The learning approach can predict the optimal orientation reliably, achieving 82% of the compression rate of the truly optimal orientation.
  • Computational Efficiency: The prediction process is orders of magnitude faster than brute-force testing, taking less than a second per clip compared to over 1.5 hours for brute-force encoding on a multi-core machine.
  • Codec Agnostic: The method works across multiple popular video codecs (H264, HEVC, VP9), demonstrating its general applicability.
  • Content-Dependent Performance: While the average reduction is significant, individual clips can see up to 77% size reduction by finding their optimal isomer.

Limitations and Future Work

  • Suboptimal Predictions: The authors acknowledge that their learning-based approach is not perfect and can sometimes select a suboptimal orientation, especially when the distribution of video sizes across orientations is multimodal.
  • Future Work: The paper suggests that the selected orientation for each Group of Pictures (GOP) could be encoded as metadata, potentially becoming a standard for 360° video. Further research could explore more sophisticated feature extraction or network architectures to improve prediction accuracy.

Summary updated

Abstract

Standard video encoders developed for conventional narrow field-of-view video are widely applied to 360deg video as well, with reasonable results. However, while this approach commits arbitrarily to a projection of the spherical frames, we observe that some orientations of a 360deg video, once projected, are more compressible than others. We introduce an approach to predict the sphere rotation that will yield the maximal compression rate. Given video clips in their original encoding, a convolutional neural network learns the association between a clip's visual content and its compressibility at different rotations of a cubemap projection. We validate our idea on thousands of video clips and multiple popular video codecs. The results show that this untapped dimension of 360deg video compression has substantial potential--"good" rotations are typically 8-10% more compressible than bad ones, and our learning approach can predict them reliably 82% of the time. The full report is published in the CVPR main conference.
BibTeX
@inproceedings{su2018learning,
  author = {Yu-Chuan Su and Kristen Grauman},
  title = {{Learning Compressible 360deg Video Isomers}},
  booktitle = {Workshops @ CVPR},
  year = {2018},
  url = {https://openreview.net/forum?id=Byb3MC-O-S},
}
Figure