Precision Enhancement of 3D Surfaces from Multiple Compressed Depth Maps

Pengfei Wan, Gene Cheung, Philip A. Chou, Dinei A. F. Florêncio, Cha Zhang, Oscar C. Au

Introduction

In the realm of 3D imaging, the "texture-plus-depth" format is a cornerstone for representing three-dimensional scenes. However, the high storage and bandwidth requirements of depth maps often necessitate lossy compression. This paper introduces a novel method to sharpen the precision of 3D depth maps that have been blurred or distorted by such compression. By treating depth maps from different viewpoints as redundant "multiple descriptions" of the same 3D scene, the authors employ an iterative projection technique—Projections onto Convex Sets (POCS)—to recover fine details lost during the quantization process.

The Challenge of Depth Map Compression

Depth maps are frequently compressed using transform coding methods like JPEG, which rely on quantization. This process rounds mathematical values into discrete "bins" to reduce data size, but it results in a significant loss of precision. The reconstructed depth map often appears as a blocky, lower-fidelity version of the original surface.

While previous research has focused on general denoising or bit-depth enhancement, this study specifically targets the distortion caused by Discrete Cosine Transform (DCT) quantization. The core challenge is to recover the "true" depth value within a quantization bin by leveraging information preserved in a different camera's viewpoint.

Methodology: Iterative Refinement via POCS

The proposed algorithm is built on the principle that the "true" depth signal must satisfy two simultaneous constraints: geometric consistency across views and adherence to the original quantization boundaries. The workflow involves an iterative process:

  1. View-to-View Projection: A depth map from a source view (e.g., "left") is back-projected into 3D space and then re-projected onto the target view's coordinate system. To maintain edge sharpness and minimize noise during this transfer, the authors utilize edge-adaptive linear interpolation and bilateral filtering.
  2. Coefficient Clipping: Once in the target view, the signal is transformed into the DCT domain. If a value falls outside the original quantization bin, it is "clipped" to the nearest boundary. This ensures the refined map remains consistent with the original compressed data.
  3. Iterative Convergence: This process repeats—projecting from left to right, then right to left—until the depth maps converge to a stable, high-precision state.

Key Results and Analysis

The researchers evaluated their approach using the New Tsukuba Stereo Dataset, a standard benchmark providing high-quality ground-truth depth maps. Success was measured using Peak Signal-to-Noise Ratio (PSNR) gain, comparing the proposed method ($Q_{our}$) against standard decoding ($Q_{std}$) and a bilateral-filtered version ($Q_{smo}$).

PSNR gain for left and right views over 20 iterations, showing significant improvement over standard and smoothed baselines.

Quantitative and Qualitative Gains

As shown in the performance graphs, the proposed method consistently outperforms both standard decoding and simple smoothing. The PSNR gain for the left view reached nearly 4 dB over the standard version after several iterations, typically converging within 10–20 cycles.

Visually, the reduction in artifacts is striking. Error maps—which visualize the difference between the ground truth and the reconstruction—demonstrate that the proposed method significantly mitigates the "blockiness" and edge errors typical of JPEG-style compression.

Error maps comparing smoothed (c) and standard (d) reconstructions to the ground truth, highlighting the artifacts the proposed method aims to eliminate.

Insights and Future Directions

The primary innovation of this work lies in treating multi-view redundancy as a tool rather than a burden. By enforcing strict quantization constraints, the algorithm prevents the "hallucination" of details, effectively using the second view to narrow down the true value within a quantization bin.

However, some limitations remain. The authors note that while the method converges empirically, a formal mathematical proof of convergence is still needed. Additionally, the iterative projections between 3D and 2D spaces are computationally intensive, presenting a challenge for real-time applications. Future work will likely focus on optimizing these computations and adapting the model for dynamic, non-static 3D scenes.

Summary updated

Abstract

Transmitting depth maps captured from multiple viewpoints of a 3-D scene enables a wide range of receiver-side 3-D applications, including virtual view synthesis via depth-image-based rendering (DIBR). Observing that compressed depth maps from different viewpoints constitute multiple descriptions (MD) of the same signal, we propose to reconstruct 3-D surfaces of the scene by considering multiple compressed depth maps jointly. Specifically, we propose an alternating projection algorithm, inspired by the theory of projection onto convex sets (POCS), which at convergence returns a 3-D surface that satisfies three sets of conditions: spatial smoothness prior, quantization bin constraints in the block transform domain, and inter-view consistency. We present a theoretical proof that shows convergence of our algorithm under benign conditions. Compared to existing multiview depth map denoising schemes and single image de-quantization schemes, our proposed solution achieves higher objective quality for both reconstructed depth maps and synthesized virtual views.
BibTeX
@article{wan2015precision,
  author = {Pengfei Wan and Gene Cheung and Philip A. Chou and Dinei A. F. Florêncio and Cha Zhang and Oscar C. Au},
  title = {{Precision Enhancement of 3D Surfaces from Multiple Compressed Depth Maps}},
  journal = {IEEE Signal Processing Letters},
  year = {2015},
  url = {https://openreview.net/forum?id=m92WZJFLHu},
}
Figure