Depth image-based rendering from multiple cameras with 3D propagation algorithm

Quang H. Nguyen, Minh N. Do, Sanjay J. Patel

Abstract

Synthesizing virtual views from multiple color and range cameras remains challenging due to disocclusions and mismatched resolutions. This paper presents a 3D propagation algorithm for Depth Image-Based Rendering (DIBR) that leverages a depth-color bilateral filter and epipole-based hole-filling. By offloading the pipeline to GPGPU, the system achieves a 277x speedup, enabling real-time synthesis at 24.37 FPS.

Introduction

Image-Based Rendering (IBR) is the art of creating "virtual" camera views from a set of "real" ones. While it offers more photorealism than traditional 3D modeling, it faces a significant hurdle: disocclusions. When a virtual camera moves, it often "sees around" objects, revealing "holes" where no data exists in the original images.

Previous solutions were often constrained to 1D setups (where all cameras sit on a single line) or assumed that every color camera had a perfectly aligned depth sensor. In reality, range (depth) cameras are often lower resolution and placed separately from high-resolution color cameras. This paper tackles the "generalized 3D case," allowing for arbitrary camera placement and mismatched resolutions while maintaining real-time speeds.

The proposed multi-camera setup involving separate color (C1, C2) and range (D1) cameras.

The 3D Propagation Algorithm

The authors propose a two-stage 3D Propagation Algorithm designed to handle the complexities of non-aligned sensors and arbitrary viewpoints.

1. Depth Propagation

Depth data from range cameras is first "warped" to the coordinate systems of the color cameras. To refine this data, the pipeline follows several critical steps:

The depth propagation pipeline, from initial warping to final disocclusion filling.
  • Occlusion Removal: This technique divides a local window into four partitions. If a pixel is significantly deeper than its neighbors in most partitions, it is flagged as a background pixel that should have been occluded and is removed.
  • Depth-Color Bilateral Filtering (DCBF): This is the core innovation. It uses the high-resolution color image to guide the interpolation of the low-resolution depth map. Since depth edges usually align with color edges, this preserves sharp boundaries.
  • Disocclusion Filling: To fill remaining holes, the algorithm calculates the "epipole" (the projection of the reference camera onto the desired view) to determine a mathematically correct filling direction, ensuring the background stretches naturally.

2. Rendering

Once each color camera has a refined depth map, the combined color and depth data are warped to the final virtual view, merged, and filtered to produce the output image.

Implementation and Performance

To make this computationally feasible, the authors mapped the 3D warping, occlusion removal, and DCBF steps onto a massively parallel GPGPU architecture.

Using an NVIDIA GeForce 9800GT, the implementation achieved 24.37 FPS, compared to just 0.09 FPS on a standard CPU (Intel Core2 Duo). This represents a 277x increase in speed, moving the technology from offline processing to real-time application. Qualitatively, the combination of occlusion removal and DCBF significantly reduced the "ghosting" and "jagged edge" artifacts common in simpler DIBR methods.

Conclusion

By treating color information as a high-fidelity proxy for depth structure and utilizing epipolar geometry for hole-filling, the 3D propagation algorithm provides a robust solution for multi-camera view synthesis. While boundary "black holes" remain a challenge due to limited camera coverage, the massive GPGPU speedup demonstrates the viability of this approach for real-time interactive systems.

Summary updated

Abstract

This paper proposes a new 3D propagation algorithm for the depth image-based rendering problem with multiple color and range cameras at arbitrary positions. The proposed algorithm efficiently renders novel images at arbitrary vir-tual views by propagating all available depth information from range cameras to color cameras, and then all avail-able depth and color information from color cameras to the virtual views. Furthermore, the algorithm significantly en-hances the propagated depth images by applying a new oc-clusion removal method and a new depth-color bilateral fil-tering. The paper also describes the parallelism structure of our algorithm and outlines a mapping onto massively par-allel architectures such as general-purpose graphics process-ing units (GPGPUs). Experimental results show that the proposed algorithm provides good rendering quality while staying within computational bounds for real-time applica-tions. Keywords Depth image-based rendering, massively parallel architec-ture, bilateral filtering 1.
BibTeX
@inproceedings{nguyen2009depth,
  author = {Quang H. Nguyen and Minh N. Do and Sanjay J. Patel},
  title = {{Depth image-based rendering from multiple cameras with 3D propagation algorithm}},
  booktitle = {IMMERSCOM},
  year = {2009},
  doi = {10.4108/immerscom.2009.13},
}
Figure