Generating High-Quality Panorama by View Synthesis Based on Optical Flow Estimation

Wenxin Zhang, Yumei Wang, Yu Liu2022 · DOI

TL;DR

This paper tackles the challenge of generating high-quality, seamless 360° panoramas by proposing a novel view synthesis approach that leverages optical flow estimation. By accurately capturing pixel-level motion between adjacent views, their method effectively reconstructs and blends images, significantly reducing artifacts and improving visual fidelity compared to traditional stitching techniques.

What Problem Does This Paper Address?

Creating immersive virtual reality (VR) experiences heavily relies on high-quality panoramic images. Traditional image stitching methods, while effective for many applications, often struggle with generating seamless 360° panoramas. These methods typically rely on global geometric transformations (like homography) which assume all overlapping scene points lie on the same depth plane. This assumption breaks down in scenes with varying depths, leading to common artifacts such as cracks, ghosting, and irregular shapes. The paper aims to overcome these limitations by shifting from traditional stitching to a view synthesis paradigm, specifically addressing the problem of generating high-quality panoramas from a set of overlapping images without requiring explicit 3D scene information or camera parameters.

Methods

The proposed method consists of two main stages:

  1. Optical Flow Estimation: This stage focuses on establishing dense, pixel-level correspondences between overlapping regions of adjacent input images (left and right views). They introduce a novel optical flow estimation algorithm that utilizes a patch-based search strategy. Key aspects include:
  2. Gray and Alpha Channel Integration: Both gray components and alpha channels are used to adjust image intensity, mitigating spatial gray differences and improving matching accuracy.
  3. Image Pyramid: Downsampling images into a pyramid structure allows for efficient processing and handling of larger motions.
  4. Extended Rectangular Search Box: To better capture foreground details and handle depth variations, the search box is expanded beyond a simple square, considering the depth variation of scene content.
  5. Gradient Information: Incorporating gradient norms in both x and y directions, along with transparency information and Gaussian blur, enhances the robustness and accuracy of the flow estimation. The optical flow is treated as an approximation of the scene's inverse depth.
  6. View Synthesis and Blending: Once the optical flow is computed, it's used to reconstruct and blend the views.
  7. Reconstruction: The left and right views are warped using the estimated optical flow. This process aims to reduce vertical distortion and compensate for parallax-induced mismatches. An amplitude adjustment parameter (δ) is introduced to control the degree of tuning based on the optical flow.
  8. Alpha Blending: A dual-weight blending algorithm is employed. It combines optical flow weights (derived from the estimated flow) with alpha weights (based on pixel transparency) to synthesize the final novel view. This approach allows for a compromise between optical flow and alpha-based blending, leading to a more robust synthesis that can tolerate parallax.

Datasets

The authors used a combination of real-world datasets for training and evaluation:

  • Custom Datasets:
    • 'scene1' and 'scene2': Captured by the Facebook Surround 360 device.
  • 'fisheye': Captured by the Insta360 device.
  • Public Datasets:
    • HCI and 'Teddy' light field datasets from Middlebury.
  • KITTI dataset (stereo pairs) for autonomous driving scenes.
  • Ablation Study Dataset:
    • Stereo/flow 2012 dataset of KITTI.

No new benchmark datasets were introduced.

Key Insights and Ideas

  • View Synthesis as a Solution: The paper convincingly argues that for panorama generation, especially for 360° content, a view synthesis approach is more suitable than traditional stitching, which struggles with depth variations.
  • Optical Flow as Inverse Depth: Treating optical flow as an approximation of inverse depth is a crucial insight, allowing the method to implicitly handle scene geometry without explicit 3D reconstruction.
  • Dual-Weight Blending: Combining optical flow-derived weights with alpha blending weights provides a more sophisticated way to handle parallax and transparency, leading to smoother transitions and fewer artifacts.
  • Handling Vertical Distortion: The reconstruction step using optical flow is shown to be effective in correcting noticeable vertical distortions, which significantly impacts the viewing experience.

Key Results

  • Subjective Improvement: Experimental results demonstrate that the proposed approach provides a better subjective visual experience compared to other methods, with fewer cracks and artifacts.
  • Objective Metrics: The method achieves significant improvements in image quality assessment metrics:
  • MP-PSNR: Approximately 1 dB improvement.
  • PSNR: Approximately 1 dB improvement.
  • SSIM: Approximately 25% improvement.
  • Ablation Study: The ablation study confirms the importance of each component: optical flow estimation (FE), flow-based reconstruction (FR), and flow-based blending (FB). The proposed method ("Ours") consistently outperforms variants that omit these components.
  • Comparison with State-of-the-Art: The proposed method shows competitive or superior performance against methods like APAP, AANAP, and SM across various datasets, particularly in handling complex textures and reducing stitching distortions.

Limitations and Future Work

  • Parallax Limitations: The authors acknowledge that for scenes with very large parallax, the optical flow adjustment might not fully overcome the resulting irregular edges, as seen in the TEDDY dataset example.
  • Computational Cost: While not explicitly detailed, optical flow estimation can be computationally intensive. The use of image pyramids helps, but further optimization might be needed for real-time applications.
  • Future Work: The paper implicitly suggests that further improvements in optical flow estimation for extreme parallax scenarios and exploring more efficient blending strategies could be future research directions. The current work focuses on static scenes; extending it to dynamic scenes with moving objects would be a natural next step.

Summary updated

BibTeX
@misc{zhang2022generating,
  author = {Wenxin Zhang and Yumei Wang and Yu Liu},
  title = {{Generating High-Quality Panorama by View Synthesis Based on Optical Flow Estimation}},
  year = {2022},
  doi = {10.3390/s22020470},
}
Figure