Novel tile segmentation scheme for omnidirectional video

Jisheng Li, Ziyu Wen, Sihan Li, Yikai Zhao, Bichuan Guo, Jiangtao Wen

TL;DR

This paper introduces a novel tile-based segmentation scheme for omnidirectional video that significantly reduces wasted pixel area and bitrate compared to traditional equirectangular projection. By intelligently dividing the spherical video into tiles, the proposed method achieves up to 28% pixel area savings and 20% BD-rate reduction, making VR content more efficient to encode and transmit.

What Problem Does This Paper Address?

The increasing popularity of virtual reality (VR) necessitates efficient methods for capturing, encoding, and transmitting omnidirectional video. Current approaches often rely on mapping the spherical video onto a 2D plane, with equirectangular projection being a common choice. However, equirectangular projection introduces significant distortions, particularly at the poles, leading to a large number of interpolated, low-information pixels. This "wasted" pixel area directly translates to increased bitrate and reduced coding efficiency. Other methods like cubic projection also have their own inefficiencies. The paper aims to address this by developing a more efficient projection and segmentation strategy that minimizes wasted pixels and improves compression.

Methods

The core of this paper is a tile-based segmentation scheme for omnidirectional video. Instead of a single, distorted 2D projection, the sphere is divided into multiple tiles. The key innovation lies in how these tiles are defined and arranged to approximate an equal-area mapping with minimal distortion.

The authors analyze the total area of tiles required for a hemisphere based on latitude cuts and overlap. They formulate an optimization problem to minimize the total projected area for a given number of tiles and overlap percentage. The scheme involves:

  • Tile Segmentation: Dividing the sphere into segments along latitude lines.
  • Projection and Rearrangement: Flattening these segments into rectangular tiles and arranging them into a single rectangular video stream for compatibility with standard decoders.
  • Overlap Handling: Introducing small overlapping regions between adjacent tiles to mitigate border artifacts that can arise from independent tile encoding, especially at low bitrates. The paper investigates the trade-off between overlap and increased total area.

The proposed scheme is compared against traditional equirectangular projection and cubic projection, as well as a prior tile-based method by Yu et al.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they used:

  • SUN360 database: 10 sequences of 9K images.
  • 4K video clips: Provided by LETINVR.

These sequences were used in their equirectangular projection format as ground truth.

Key Insights and Ideas

  • The "wasted pixel" problem is significant: Equirectangular projection can introduce up to 57% more pixel area than the original sphere, directly impacting bitrate.
  • Intelligent tiling is key: By dividing the sphere into carefully defined tiles, it's possible to achieve a much more efficient representation that closely approximates equal-area mapping.
  • Overlap is a necessary evil: While introducing overlap between tiles helps with compression artifacts, it also increases the total area. The paper quantifies this trade-off, showing that even a 0.5% overlap (around 10 pixels for 4K video) incurs a minimal bitrate overhead (under 1.7%).
  • The number of tiles matters: Increasing the number of tiles generally reduces the total area, but there's a practical limit due to encoding overhead and decoder complexity.
  • L-PSNR is a more relevant metric: The paper highlights the importance of L-PSNR (latitude-weighted PSNR) over standard PSNR for evaluating omnidirectional video quality, as it better accounts for the varying perceptual importance of different regions on the sphere.

Key Results

  • The proposed tile segmentation scheme can save up to 28% of pixel area compared to equirectangular projection.
  • It achieves an average 20% BD-rate saving compared to traditional equirectangular projection.
  • Compared to a prior tile-based method (Yu et al.), the proposed scheme offers an additional 4-6% bitrate saving.
  • Introducing a 0.5% overlap between tiles results in no more than 1.7% bitrate overhead on average.
  • The paper demonstrates that the proposed 3-tile and 5-tile schemes (P3 and P5) outperform cubic projection (CP) and Yu's 3-tile and 5-tile schemes (Y3 and Y5) in terms of BD-rate savings.

Limitations and Future Work

  • Decoder Compatibility: While the paper mentions rearranging tiles into a single rectangular stream for compatibility, the overhead associated with this rearrangement is not fully detailed.
  • Overhead of Tile-Based Encoding: The paper acknowledges that using many tiles introduces overhead due to independent encoding of each tile. The optimal number of tiles is a balance between area saving and this overhead.
  • Future Work: The authors suggest future research directions including:
  • Optimizing the segmentation scheme for region of interest (ROI) applications.
  • Developing optimizations for temporal and spatial random access.

Summary updated

Abstract

Regular omnidirectional video encoding technics use map projection to flatten a scene from a spherical shape into one or several 2D shapes. Common projection methods including equirectangular and cubic projection have varying levels of interpolation that create a large number of non-information-carrying pixels that lead to wasted bitrate. In this paper, we propose a tile based omnidirectional video segmentation scheme which can save up to 28% of pixel area and 20% of BD-rate averagely compared to the traditional equirectangular projection based approach.
BibTeX
@inproceedings{li2016novel,
  author = {Jisheng Li and Ziyu Wen and Sihan Li and Yikai Zhao and Bichuan Guo and Jiangtao Wen},
  title = {{Novel tile segmentation scheme for omnidirectional video}},
  booktitle = {ICIP},
  year = {2016},
  doi = {10.1109/icip.2016.7532381},
}
Figure