Hole Filling With Multiple Reference Views in DIBR View Synthesis

Shuai Li, Ce Zhu, Ming-Ting Sun

TL;DR

This paper proposes a more accurate way to fill "holes" in synthesized 3D video views by using additional reference cameras rather than just guessing missing pixels. By introducing a "selective warping" scheme, the authors significantly reduce hole size and improve image quality with minimal computational overhead.

What Problem Does This Paper Address?

In 3D video systems, Depth-Image-Based Rendering (DIBR) allows us to synthesize a virtual "middle" view from existing camera angles. However, a major challenge is disocclusion: when a foreground object moves, it reveals background areas that were hidden in the original reference views. These missing areas appear as "holes."

Traditional methods fill these holes using inpainting (extrapolating or interpolating from neighboring pixels). This works fine for simple backgrounds but fails miserably with complex textures, leading to blurry or erroneous artifacts. The authors argue that instead of guessing what the background looks like, we should look for that information in other available camera views that might have had a clear line of sight to those occluded areas.

Methods

The authors move beyond the standard "two-primary-view" setup for interpolation and "one-view" setup for extrapolation. Their approach involves two main components:

  1. Multiple Reference View Framework: They analytically examine how holes are generated and demonstrate that "complementary views" (cameras further away from the virtual viewpoint) often contain the exact pixels needed to fill disocclusions.
  2. Selective Warping Scheme: Warping an entire high-resolution frame from a third or fourth camera is computationally expensive and wasteful, as only a few pixels are actually needed to fill holes. The authors developed a "backward warping" technique:
  3. They identify the edge of a hole in the virtual view.
  4. They estimate the background depth at that edge.
  5. They "look back" into the complementary view to locate only the specific pixels required to fill that specific hole.
  6. This avoids a full 3D warping of the additional views, keeping the system efficient.

Datasets

The researchers evaluated their framework using several well-known multi-view video sequences:

  • Ballet and Breakdancers: Standard 1024x768 sequences.
  • UndoDancer, Shark, and Bee: Higher resolution 1920x1088 sequences.
  • Shark is specifically noted as an FTV (Free-viewpoint TV) sequence, while others are common in 3D Video (3DV) research.

Key Insights and Ideas

  • From Guessing to Finding: The core "aha" moment is treating hole-filling as a data retrieval problem rather than a signal processing estimation problem. If the data exists in another view, using it will always be superior to even the best inpainting algorithm.
  • The Geometry of Holes: The paper provides a rigorous mathematical breakdown of hole locations based on foreground/background segments. This allows them to predict exactly which complementary views will be useful.
  • Efficiency through Sparsity: By recognizing that holes occupy a tiny fraction of the total image area, they realized they could skip 88% of the warping work usually required for additional views.

Key Results

  • Hole Reduction: In view interpolation, using two extra complementary views reduced the hole size by up to 70%. In view extrapolation, one extra view reduced hole size by about 27%.
  • Visual Quality: The Peak Signal-to-Noise Ratio (PSNR) of the filled pixels was significantly higher than standard inpainting. For the Ballet sequence, the proposed method was more than 2 dB better than full warping and even higher compared to standard VSRS inpainting.
  • Computational Speed: While full warping of extra views doubled the rendering time, the selective warping scheme only added about 12% to the total processing time.

Limitations and Future Work

  • Depth Map Dependency: The method relies heavily on the accuracy of depth maps. In the Breakdancers sequence, poor depth map quality led to less impressive hole reduction because the "backward warp" pointed to the wrong locations in the reference views.
  • Boundary Noise: The authors noted "boundary artifacts" or "contour noise" caused by color bleeding at the edges of foreground objects in the reference views. While their method fills the holes, it doesn't entirely eliminate the noisy "halo" effect often seen in DIBR.
  • Future Directions: The authors suggest that future work could integrate better depth estimation or specific filters to handle the boundary noise that persists even when holes are correctly filled.

Summary updated

Abstract

Depth-image-based rendering (DIBR) oriented view synthesis has been widely employed in the current depth-based 3-D video systems by synthesizing a virtual view from an arbitrary viewpoint. However, holes may appear in the synthesized view due to disocclusion, thus significantly degrading the quality. Consequently, efforts have been made on developing effective and efficient hole-filling algorithms. Current hole-filling techniques generally extrapolate/interpolate the hole regions with the neighboring information based on an assumption that the texture pattern in the holes is similar to that of the neighboring background information. However, in many scenarios, especially of complex texture, the assumption may not hold. In other words, hole-filling techniques can only provide an estimation for a hole which may not be good enough or may even be erroneous considering a wide variety of complex scene of images. In this paper, we first examine the view interpolation with multiple reference views, demonstrating that the problem of emerging holes in a target virtual view can be greatly alleviated by making good use of other neighboring complementary views in addition to its two (commonly used) most neighboring primary views. The effects of using multiple views for view extrapolation in reducing holes are also investigated in this paper. In view of the 3D Video and ongoing free-viewpoint TV standardization, we propose a new view synthesis framework, which employs multiple views to synthesize output virtual views. Furthermore, a scheme of selective warping of complementary views is developed by efficiently locating a small number of useful pixels in the complementary views for hole reduction, to avoid full warping of additional complementary views thus lowering greatly the warping complexity. Experimental results show that the hole size based on two primary reference views may be reduced by up to about 70% with the help of two complementary reference views in the case of view interpolation, while the hole size based on one primary reference view may be reduced by about 27% with the help of one more complementary reference view in view extrapolation. Moreover, it is shown that by using one more pair of views in view interpolation and one more view in view extrapolation, 10% hole pixels may be reduced additionally.
BibTeX
@article{li2018hole,
  author = {Shuai Li and Ce Zhu and Ming-Ting Sun},
  title = {{Hole Filling With Multiple Reference Views in DIBR View Synthesis}},
  journal = {IEEE Trans. Multimedia},
  year = {2018},
  url = {https://openreview.net/forum?id=grfdNgW2Cc3h},
}

Loading related content...

Figure