Hole Filling with Multiple Reference Views in DIBR View Synthesis

Shuai Li, Ce Zhu, Ming-Ting Sun

Abstract

Depth-image-based rendering (DIBR) is a cornerstone of modern 3D video systems, enabling the synthesis of virtual views from arbitrary viewpoints. However, the process frequently creates "holes" due to disocclusion—areas where foreground objects move to reveal previously hidden background. Traditional hole-filling methods rely on inpainting, which estimates missing pixels based on neighboring textures. This paper proposes a more robust framework that utilizes multiple reference views to retrieve actual pixel data for these holes. By introducing a "selective warping" scheme, the authors achieve significant hole reduction and improved visual quality with minimal computational overhead.

The Challenge of Disocclusion

In DIBR, a virtual view is typically synthesized from two neighboring "primary" views. When the virtual viewpoint shifts, foreground objects and backgrounds move at different rates, leaving empty regions (holes) where no data is available from the primary cameras.

Standard techniques fill these gaps using spatial inpainting. While effective for simple, uniform backgrounds, inpainting often fails in complex scenes, producing blurry or visually incorrect artifacts. The core premise of this research is that these "missing" pixels are often not missing at all—they are simply occluded in the primary views but visible from other, more distant "complementary" cameras in a multi-view setup.

Methodology: Selective Warping

To utilize these additional views without doubling the computational cost of the rendering pipeline, the authors developed a selective warping scheme. Instead of performing a full 3D warping of every additional camera view, the system identifies specific hole regions in the virtual view and "looks back" into the complementary views to find the necessary data.

The selective warping mechanism uses backward warping to identify and fetch only the specific background pixels needed from complementary views to fill holes.

This "backward warping" technique involves:

  1. Identifying the edges of a hole in the synthesized virtual view.
  2. Estimating the background depth at those edges.
  3. Projecting those specific coordinates back into the complementary reference views to retrieve the true texture and depth information.

This targeted approach allows the system to fill holes with high accuracy while avoiding the processing of millions of unnecessary pixels.

Key Results and Performance

The framework was tested on several high-resolution multi-view sequences, including Ballet, Shark, and UndoDancer. The results demonstrate a dramatic reduction in hole size and a measurable increase in image fidelity.

Hole size comparison for the Ballet sequence shows that the proposed 2PV+2CV (two primary plus two complementary views) method significantly reduces the number of missing pixels compared to the standard two-view (2PV) approach.
  • Hole Reduction: In view interpolation, using two additional complementary views reduced the total hole area by up to 70%.
  • Visual Fidelity: The Peak Signal-to-Noise Ratio (PSNR) of the filled regions improved by more than 2 dB compared to standard full-warping methods in the Ballet sequence.
  • Efficiency: While full warping of additional views typically doubles the rendering time, the selective warping scheme added only about 12% to the total processing time, making it viable for real-time applications.

Conclusion

By shifting the hole-filling problem from one of signal estimation (inpainting) to one of data retrieval (multi-view warping), this framework provides a more accurate representation of occluded backgrounds. The selective warping scheme ensures that this accuracy does not come at the cost of prohibitive computational complexity. Future work may focus on refining depth map accuracy and addressing boundary noise to further polish the synthesized output.

Summary updated

Abstract

Depth-image-based rendering (DIBR) oriented view synthesis has been widely employed in the current depth-based 3D video systems by synthesizing a virtual view from an arbitrary viewpoint. However, holes may appear in the synthesized view due to disocclusion, thus significantly degrading the quality. Consequently, efforts have been made on developing effective and efficient hole filling algorithms. Current hole filling techniques generally extrapolate/interpolate the hole regions with the neighboring information based on an assumption that the texture pattern in the holes is similar to that of the neighboring background information. However, in many scenarios especially of complex texture, the assumption may not hold. In other words, hole filling techniques can only provide an estimation for a hole which may not be good enough or may even be erroneous considering a wide variety of complex scene of images. In this paper, we first examine the view interpolation with multiple reference views, demonstrating that the problem of emerging holes in a target virtual view can be greatly alleviated by making good use of other neighboring complementary views in addition to its two (commonly used) most neighboring primary views. The effects of using multiple views for view extrapolation in reducing holes are also investigated in this paper. In view of the 3D Video and ongoing free-viewpoint TV standardization, we propose a new view synthesis framework which employs multiple views to synthesize output virtual views. Furthermore, a scheme of selective warping of complementary views is developed by efficiently locating a small number of useful pixels in the complementary views for hole reduction, to avoid a full warping of additional complementary views thus lowering greatly the warping complexity.
BibTeX
@article{li2018hole,
  author = {Shuai Li and Ce Zhu and Ming-Ting Sun},
  title = {{Hole Filling with Multiple Reference Views in DIBR View Synthesis}},
  journal = {IEEE Trans. Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=grfdNgW2Cc3h},
}
Figure