Abstract
Synthesizing virtual views from multiple color and range cameras remains challenging due to disocclusions and mismatched resolutions. This paper presents a 3D propagation algorithm for Depth Image-Based Rendering (DIBR) that leverages a depth-color bilateral filter and epipole-based hole-filling. By offloading the pipeline to GPGPU, the system achieves a 277x speedup, enabling real-time synthesis at 24.37 FPS.
Introduction
Image-Based Rendering (IBR) is the art of creating "virtual" camera views from a set of "real" ones. While it offers more photorealism than traditional 3D modeling, it faces a significant hurdle: disocclusions. When a virtual camera moves, it often "sees around" objects, revealing "holes" where no data exists in the original images.
Previous solutions were often constrained to 1D setups (where all cameras sit on a single line) or assumed that every color camera had a perfectly aligned depth sensor. In reality, range (depth) cameras are often lower resolution and placed separately from high-resolution color cameras. This paper tackles the "generalized 3D case," allowing for arbitrary camera placement and mismatched resolutions while maintaining real-time speeds.
The 3D Propagation Algorithm
The authors propose a two-stage 3D Propagation Algorithm designed to handle the complexities of non-aligned sensors and arbitrary viewpoints.
1. Depth Propagation
Depth data from range cameras is first "warped" to the coordinate systems of the color cameras. To refine this data, the pipeline follows several critical steps:
- Occlusion Removal: This technique divides a local window into four partitions. If a pixel is significantly deeper than its neighbors in most partitions, it is flagged as a background pixel that should have been occluded and is removed.
- Depth-Color Bilateral Filtering (DCBF): This is the core innovation. It uses the high-resolution color image to guide the interpolation of the low-resolution depth map. Since depth edges usually align with color edges, this preserves sharp boundaries.
- Disocclusion Filling: To fill remaining holes, the algorithm calculates the "epipole" (the projection of the reference camera onto the desired view) to determine a mathematically correct filling direction, ensuring the background stretches naturally.
2. Rendering
Once each color camera has a refined depth map, the combined color and depth data are warped to the final virtual view, merged, and filtered to produce the output image.
Implementation and Performance
To make this computationally feasible, the authors mapped the 3D warping, occlusion removal, and DCBF steps onto a massively parallel GPGPU architecture.
Using an NVIDIA GeForce 9800GT, the implementation achieved 24.37 FPS, compared to just 0.09 FPS on a standard CPU (Intel Core2 Duo). This represents a 277x increase in speed, moving the technology from offline processing to real-time application. Qualitatively, the combination of occlusion removal and DCBF significantly reduced the "ghosting" and "jagged edge" artifacts common in simpler DIBR methods.
Conclusion
By treating color information as a high-fidelity proxy for depth structure and utilizing epipolar geometry for hole-filling, the 3D propagation algorithm provides a robust solution for multi-camera view synthesis. While boundary "black holes" remain a challenge due to limited camera coverage, the massive GPGPU speedup demonstrates the viability of this approach for real-time interactive systems.
Summary updated