Robust Multi-homography Method for Image Stitching under Large Viewpoint Changes

Wuxia Yan, Chuancai Liu, Furong PengInternational Journal of Hybrid Information Technology 2017 · DOI

TL;DR

This paper introduces a novel image stitching method that tackles the challenge of creating seamless panoramas from images with significant viewpoint changes, particularly for scenes with both buildings and ground. By segmenting images into distinct parts (building and ground) and applying different homographies to each, then blending them with Gaussian weights, the method achieves more accurate alignment and natural-looking results compared to traditional single-homography approaches.

What Problem Does This Paper Address?

Image stitching, the process of combining multiple images into a single panorama, is widely used but traditionally struggles when the input images have significant parallax (i.e., large differences in viewpoint). This is especially problematic for scenes containing both vertical structures like buildings and horizontal surfaces like the ground. Standard methods often rely on a single homography (a geometric transformation) to align images, which can lead to severe distortions, ghosting, or broken structures when viewpoints change drastically. The paper specifically addresses the difficulty of stitching building images with ground under large viewpoint changes, where traditional methods fail to preserve the structure of both elements simultaneously.

Methods

The core of the proposed method is a robust multi-homography image composition technique. The key steps are:

  • Content-based Image Segmentation: Instead of relying solely on feature matching, the paper first segments the image into distinct parts based on their content. For building-ground scenes, this means separating the building section from the ground section. This is achieved by using Hough transformation to detect parallel lines in architectural scenes, which are then used to define the boundary between the building and ground.
  • Affine-Invariant Feature Extraction: Different feature descriptors are employed for different image parts. For the textured building parts, SIFT is used. For the smoother ground parts, which are more challenging for feature extraction, ASIFT is chosen due to its robustness to viewpoint changes and its superior performance in experiments.
  • Multi-homography Calculation: Once segmented, separate homographies are calculated for each part of the image using the extracted features. This allows for more accurate geometric transformations tailored to the specific geometry of each part.
  • Weighted Warping and Blending: The individual homographies are then blended using Gaussian weights. This weighting is based on the distance of pixels from the segmented boundary, assigning higher weights to pixels closer to the boundary and lower weights to those further away. This weighted blending helps to regularize the process and reduce projective distortion, creating a more natural panorama. The method is also generalized to handle images with more than two predominant parts.

Datasets

The paper evaluates its method using a range of input images, specifically focusing on building images with ground under large viewpoint changes. These images include various types of ground surfaces (smoothed, shadowed, lawn, floor tiles) and buildings (with rich texture or smoothed walls). The paper does not introduce a new benchmark dataset but uses these diverse real-world image pairs to demonstrate the effectiveness of their approach.

Key Insights and Ideas

  • Segmentation is Crucial for Complex Scenes: The paper highlights that simply applying a single global transformation is insufficient for scenes with distinct geometric elements (like buildings and ground) under large viewpoint changes. Content-based segmentation is a critical first step to enable specialized handling of different scene parts.
  • Feature Descriptor Choice Matters: The performance of feature extraction on smooth surfaces like ground is significantly poorer than on textured surfaces. The paper demonstrates that ASIFT is a more suitable descriptor than SIFT for these challenging ground regions, leading to better feature matching and overall alignment.
  • Multi-homography for Local Accuracy: By calculating and blending multiple homographies, the method can better preserve the local geometric structure of different image parts, leading to a more accurate and visually pleasing panorama compared to global methods.
  • Gaussian Weighting for Smooth Transitions: The use of Gaussian weights for blending the different homographies is key to creating seamless transitions and mitigating distortions at the boundaries between segmented parts.

Key Results

The experimental results demonstrate that the proposed multi-homography method achieves:

  • Higher Alignment Accuracy: The method produces more accurate alignment, particularly in the ground and lower building sections, where traditional methods often exhibit misalignment.
  • More Natural Panoramas: The resulting panoramas are visually more natural and exhibit fewer distortions and artifacts compared to traditional projective transformation methods and other state-of-the-art techniques like Moving DLT and SPHP.
  • Robustness to Large Viewpoint Changes: The method is shown to be robust in handling images with significant viewpoint changes, a common challenge in image stitching.
  • Performance: The proposed method typically takes 8 to 13 seconds to stitch images on a PC with a 2.93 GHz CPU and 2GB RAM.

Limitations and Future Work

  • Limitations: The paper primarily focuses on scenes with a clear distinction between a building part at the top and a ground part at the bottom. While it mentions extending to images with more than two predominant parts, the detailed experimental validation is concentrated on the building-ground scenario. The performance on extremely smooth or textureless ground surfaces might still be challenging.
  • Future Work: The authors suggest that the method can be extended to stitch images with largely different blocks, implying potential applications beyond just building-ground scenes. Further research could explore more sophisticated segmentation techniques or adaptive weighting strategies for even greater robustness and generality.

Summary updated

BibTeX
@article{yan2017robust,
  author = {Wuxia Yan and Chuancai Liu and Furong Peng},
  title = {{Robust Multi-homography Method for Image Stitching under Large Viewpoint Changes}},
  journal = {International Journal of Hybrid Information Technology},
  year = {2017},
  doi = {10.14257/ijhit.2017.10.9.01},
}
Figure