Virtual View Synthesis Based on DIBR and Image Inpainting

Yuhan Gao, Hui Chen, Weisong Gao, Tobi VaudreyLecture Notes in Computer Science 2014 · DOI · OpenAlex

Abstract

Virtual View Synthesis is a critical technology for 3DTV and Virtual Reality, enabling the generation of novel viewpoints from a limited set of reference cameras. This paper presents a refined pipeline that integrates Depth-Image-Based-Rendering (DIBR) with morphological filtering and depth-aware image inpainting. The proposed approach specifically targets the mitigation of disocclusion "holes" and "edge-ghosting" artifacts. By utilizing a novel image-blending heuristic and morphological closing operations on depth maps, the system achieves a Peak Signal-to-Noise Ratio (PSNR) of 33.7350, significantly outperforming traditional DIBR methods.

Introduction

Depth-Image-Based-Rendering (DIBR) is the standard technique for synthesizing virtual views in 3D video systems. Given a reference color image and its corresponding depth map, DIBR warps pixels to a new camera position. However, this process inherently creates artifacts. "Holes" appear in regions occluded in the reference view but visible in the virtual view. Furthermore, misalignments between depth and color data often lead to "edge-ghosting," where foreground textures bleed into the background.

Proposed Methodology

The authors introduce a multi-stage pipeline designed for efficiency and high visual fidelity. Unlike traditional methods that rely on computationally expensive depth-map pre-processing, this pipeline focuses on intelligent blending and post-warping refinement.

The proposed virtual view synthesis pipeline.

1. Improved Image Blending

The process begins by warping raw images from two reference views (left and right). A specific heuristic is applied during blending to decide which source data to prioritize. If a pixel is a hole in one view but valid in the other, the valid data is used, reducing the initial hole count before further processing.

2. Morphological Depth Filtering

To clean the resulting depth map, the authors employ a morphological "closing" operation—a combination of dilation and erosion—using a diamond-shaped window. This technique is particularly effective at filling "lump" holes (large contiguous missing regions) that standard median or bilateral filters often fail to resolve.

Comparison of depth map filtering: (a) original depth image, (b) Mori et al. method, and (c) the presented morphological filtering approach.

3. Depth-Aware Inpainting and Interpolation

Remaining holes are filled using a depth-based inpainting algorithm. By weighting neighboring pixels based on depth similarity, the system prevents foreground colors from leaking into background holes. To eliminate edge-ghosting, a Canny edge detector identifies boundaries; pixels with sharp depth/color discrepancies are replaced by an average of neighbors at a similar depth level.

4. Post-Processing

Finally, Speckle Reducing Anisotropic Diffusion (SRAD) is applied to remove tiny spots while preserving sharp edges, ensuring a clean final output.

Experimental Results

The algorithm was evaluated using the "breakdancers" sequence (1024 × 768 resolution). Cameras 2 and 4 served as references to synthesize virtual camera 3, with the actual footage from camera 3 used as ground truth.

The proposed pipeline achieved a PSNR of 33.7350 and an SSIM of 0.9345. In comparison, traditional DIBR synthesis reached only 30.4286 PSNR. Ablation studies indicated that depth-based inpainting provided the most significant quality boost (+2.1 PSNR), followed by edge-ghosting elimination via interpolation (+0.7 PSNR) and post-filtering (+0.4 PSNR).

Conclusion

This paper demonstrates that moving complexity from pre-processing to the blending and inpainting stages can yield superior virtual view quality. The use of morphological closing for depth maps and depth-aware interpolation effectively addresses the primary artifacts of DIBR. Future work will focus on extending this framework to handle more complex camera geometries and view extrapolation.

Summary updated

BibTeX
@inproceedings{gao2014virtual,
  author = {Yuhan Gao and Hui Chen and Weisong Gao and Tobi Vaudrey},
  title = {{Virtual View Synthesis Based on DIBR and Image Inpainting}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2014},
  doi = {10.1007/978-3-642-53842-1\_15},
}
Figure