Depth map coding with elastic contours and 3D surface prediction

Marco Calemme, Pietro Zanuttigh, Simone Milani, Marco Cagnazzo, Béatrice Pesquet-Popescu

TL;DR

This paper introduces a novel depth map coding scheme that significantly improves compression efficiency by representing depth data using contours and predicting 3D surfaces. This approach outperforms HEVC at medium-low bitrates, preserving sharp edges crucial for applications like view synthesis.

What Problem Does This Paper Address?

Depth maps, essential for 3D applications like virtual reality and augmented reality, are often represented as grayscale images and compressed using standard image codecs. However, this approach fails to leverage the unique characteristics of depth data, particularly the presence of smooth regions separated by sharp edges. Existing methods struggle to efficiently represent these contours and propagate temporal information, leading to suboptimal compression and loss of critical edge detail. This paper aims to bridge this gap by developing a specialized depth map coding scheme that explicitly models and compresses contours while efficiently predicting the underlying 3D surfaces.

Methods

The proposed approach breaks down depth map compression into several key stages:

  • Segmentation: The depth map is first segmented into regions, identifying smooth areas and sharp boundaries.
  • Contour Coding: A novel contour coding scheme based on "elastic curves" is employed. This method exploits temporal consistency across frames to compactly represent the contours, allowing for efficient interpolation and prediction of their evolution.
  • Subsampling and Average Depth: The depth map is subsampled, and the average depth value within each segment is computed.
  • Differential Prediction and HEVC Coding: The subsampled data and segment averages are then compressed using a differential prediction scheme followed by the HEVC coder. This step focuses on efficiently encoding the low-resolution depth information.
  • 3D Surface Prediction: A 3D surface prediction algorithm, derived from existing work, is used to reconstruct an accurate depth field from the compressed contours and subsampled data.
  • Residual Compression: Finally, the residuals (differences between the predicted and actual depth maps) are lossy compressed using the HEVC coder.

The novelty lies in the elastic contour coding and the integrated approach of using contours and subsampled data for accurate surface prediction, leading to efficient compression and edge preservation.

Datasets

The paper evaluates its approach on three sequences:

  • "undo dancer" (high-resolution synthetic)
  • "lovebird" (real-world sequence)
  • "mobile" (synthetic sequence)

These sequences are used for comparison against HEVC and a prior segmentation-based depth coding approach [4]. No new benchmarks are introduced.

Key Insights and Ideas

  • Elastic Contours for Temporal Consistency: The use of elastic curves to model contour evolution across frames is a powerful way to exploit temporal redundancy, leading to more efficient contour coding than traditional methods.
  • Contour-Driven Surface Prediction: By explicitly using contours to guide the 3D surface prediction, the method can accurately reconstruct depth fields, especially around sharp edges, which is crucial for applications like view warping.
  • Edge Preservation at Low Bitrates: The proposed scheme demonstrates a significant advantage in preserving sharp edges even at low bitrates, a common failure point for general-purpose image codecs like HEVC.
  • Synergy of Segmentation and Prediction: The combination of segmenting the depth map, coding its contours efficiently, and then using these contours to guide a surface prediction algorithm creates a highly effective compression pipeline.

Key Results

  • The proposed approach outperforms the HEVC coder at medium-low bitrates, achieving performance gains of up to 2 dB on the "undo dancer" sequence at around 0.006 bpp.
  • It significantly outperforms the segmentation-based depth coding approach of [4] across all tested bitrates.
  • The method effectively preserves sharp contours, which remain sharp across the entire bitrate range, unlike HEVC which only achieves this at higher bitrates.
  • For the "mobile" sequence, where edges are already blurred, the approach still outperforms HEVC at low bitrates, though the gains are less pronounced due to the inherent difficulty in preserving already degraded edges.
  • Qualitative results show better interpolation and fewer artifacts around people's edges when using depth compressed with the proposed method compared to HEVC.

Limitations and Future Work

  • Segmentation Dependency: The performance of the approach is tied to the quality of the initial segmentation. The authors acknowledge that the segmentation masks can have a significant impact on the total bitrate.
  • Future Work: The authors suggest replacing segmentation masks with edge-based schemes that can handle partial and interrupted edges. They also aim for better exploitation of temporal redundancy.

Summary updated

Abstract

Depth maps are typically made of smooth regions separated by sharp edges. Following this rationale, this paper presents a novel coding scheme where depth data is represented by a set of contours defining the various regions together with a compact representation of the values inside each region. The proposed coding scheme is based on elastic curves, which make possible to compactly represent the contours exploiting also the temporal consistency in different frames. A 3D surface prediction algorithm is then used to obtain an accurate estimation of the depth field from the coded contours and a subsampled version of the data. Finally, an ad-hoc coding strategy for the low resolution data and the prediction residuals is presented. Experimental results prove how the proposed approach is able to obtain a very high coding efficiency outperforming the HEVC coder at medium-low bitrates.
BibTeX
@inproceedings{calemme2016depth,
  author = {Marco Calemme and Pietro Zanuttigh and Simone Milani and Marco Cagnazzo and Béatrice Pesquet-Popescu},
  title = {{Depth map coding with elastic contours and 3D surface prediction}},
  booktitle = {ICIP},
  year = {2016},
  doi = {10.1109/icip.2016.7532529},
}
Figure