Abstract
Depth-image-based rendering (DIBR) is a cornerstone of modern 3D video systems, enabling the synthesis of virtual views from arbitrary viewpoints. However, the process frequently creates "holes" due to disocclusion—areas where foreground objects move to reveal previously hidden background. Traditional hole-filling methods rely on inpainting, which estimates missing pixels based on neighboring textures. This paper proposes a more robust framework that utilizes multiple reference views to retrieve actual pixel data for these holes. By introducing a "selective warping" scheme, the authors achieve significant hole reduction and improved visual quality with minimal computational overhead.
The Challenge of Disocclusion
In DIBR, a virtual view is typically synthesized from two neighboring "primary" views. When the virtual viewpoint shifts, foreground objects and backgrounds move at different rates, leaving empty regions (holes) where no data is available from the primary cameras.
Standard techniques fill these gaps using spatial inpainting. While effective for simple, uniform backgrounds, inpainting often fails in complex scenes, producing blurry or visually incorrect artifacts. The core premise of this research is that these "missing" pixels are often not missing at all—they are simply occluded in the primary views but visible from other, more distant "complementary" cameras in a multi-view setup.
Methodology: Selective Warping
To utilize these additional views without doubling the computational cost of the rendering pipeline, the authors developed a selective warping scheme. Instead of performing a full 3D warping of every additional camera view, the system identifies specific hole regions in the virtual view and "looks back" into the complementary views to find the necessary data.
This "backward warping" technique involves:
- Identifying the edges of a hole in the synthesized virtual view.
- Estimating the background depth at those edges.
- Projecting those specific coordinates back into the complementary reference views to retrieve the true texture and depth information.
This targeted approach allows the system to fill holes with high accuracy while avoiding the processing of millions of unnecessary pixels.
Key Results and Performance
The framework was tested on several high-resolution multi-view sequences, including Ballet, Shark, and UndoDancer. The results demonstrate a dramatic reduction in hole size and a measurable increase in image fidelity.
- Hole Reduction: In view interpolation, using two additional complementary views reduced the total hole area by up to 70%.
- Visual Fidelity: The Peak Signal-to-Noise Ratio (PSNR) of the filled regions improved by more than 2 dB compared to standard full-warping methods in the Ballet sequence.
- Efficiency: While full warping of additional views typically doubles the rendering time, the selective warping scheme added only about 12% to the total processing time, making it viable for real-time applications.
Conclusion
By shifting the hole-filling problem from one of signal estimation (inpainting) to one of data retrieval (multi-view warping), this framework provides a more accurate representation of occluded backgrounds. The selective warping scheme ensures that this accuracy does not come at the cost of prohibitive computational complexity. Future work may focus on refining depth map accuracy and addressing boundary noise to further polish the synthesized output.
Summary updated