Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS

Haoze Wu, Chun Bao, Qun Hao, Jie Cao, Li ZhangSensors 2024 · DOI

TL;DR

This paper introduces SuperUDIS, an improved unsupervised image stitching algorithm that combines state-of-the-art feature extraction (Superpoint) and matching (Lightglue) with a refined fusion process. SuperUDIS significantly enhances stitching quality, particularly for complex scenes with linear structures and color variations, outperforming existing methods like UDIS.

What Problem Does This Paper Address?

Capturing wide field-of-view (FOV) images is crucial for many applications, but hardware limitations often restrict individual image FOVs. Image stitching aims to overcome this by merging multiple images into a larger, seamless panorama. However, existing stitching methods, especially unsupervised deep learning approaches like UDIS, struggle with complex environments. These challenges include distorted linear structures, low resolution in stitched outputs, and unnatural color transitions at seams, particularly when dealing with significant color differences or intricate multi-linear patterns. This paper seeks to address these limitations by proposing a more robust and accurate unsupervised stitching algorithm.

Methods

The SuperUDIS algorithm builds upon the Unsupervised Deep Image Stitching (UDIS) framework by incorporating several key improvements:

  • Enhanced Feature Extraction and Matching: Instead of relying on traditional feature detectors or UDIS's internal methods, SuperUDIS leverages the powerful Superpoint algorithm for accurate and efficient feature point extraction and Lightglue for robust feature matching. This combination is designed to improve the precision of image registration, even in challenging scenarios with limited overlap.
  • Chroma Balance Algorithm: To address the issue of unnatural color transitions at stitching seams, especially in images with large color differences, a novel chroma balance algorithm is introduced. This algorithm adjusts the gain coefficients of overlapping image regions to ensure smoother color integration.
  • Optimized Loss Function for Fusion: The UDIS fusion algorithm's loss function is modified to improve the continuity of structural edges. Specifically, the first-order differential operators used to measure smoothness are replaced with a second-order differential Laplacian operator. This change emphasizes structural edges more effectively, leading to better preservation of linear structures and reduced distortion.

These components are integrated to create the SuperUDIS algorithm, which aims to provide superior stitching quality across a wider range of complex environments.

Datasets

The paper utilizes the following datasets for training and evaluation:

  • HPatches: Used for image registration experiments to evaluate feature extraction and matching performance.
  • SEAGULL dataset: Used for image fusion experiments to assess the overall stitching quality.
  • LPC dataset: Also used for image fusion experiments, similar to the SEAGULL dataset.

Key Insights and Ideas

  • Synergy of Superpoint and Lightglue: The paper highlights that combining Superpoint for feature detection and Lightglue for matching significantly improves image registration accuracy compared to traditional methods and the original UDIS approach, laying a stronger foundation for subsequent fusion.
  • Laplacian for Structural Integrity: Replacing first-order differentials with a second-order Laplacian operator in the loss function is a crucial insight for preserving linear structures. This approach effectively penalizes deviations at structural edges, leading to more geometrically accurate panoramas.
  • Chroma Balance for Seamlessness: The introduction of a dedicated chroma balance algorithm directly tackles the problem of jarring color shifts at seams, a common artifact in stitched images, thereby enhancing visual appeal.
  • Unsupervised Approach Advantage: By building upon an unsupervised framework (UDIS), the proposed SuperUDIS method avoids the need for labeled data, making it more broadly applicable.

Key Results

  • Quantitative Improvements: SuperUDIS achieved an average increase of 0.5 in PSNR and 0.02 in SSIM compared to the UDIS algorithm on the SEAGULL and LPC datasets.
  • Registration Accuracy: The Superpoint + Lightglue combination demonstrated significantly lower H-matrix estimation errors on the HPatches dataset compared to ORB + RANSAC and UDIS.
  • Qualitative Superiority: Visual comparisons show that SuperUDIS effectively resolves issues like stitching dislocation, linear structure distortion, and color discontinuity that persist in UDIS and other methods, particularly in complex scenes. The algorithm is noted to be more robust in environments with large color differences or multi-linear structures.

Limitations and Future Work

  • Limitations: While SuperUDIS shows significant improvements, the paper acknowledges that the UDIS fusion algorithm, even with improvements, can still face challenges in some complex situations. The ablation studies also indicate that while each component contributes positively, their combined effect is most pronounced.
  • Future Work: The paper suggests that the SuperUDIS algorithm provides a strong foundation for future advancements in image stitching. The successful integration of Superpoint, Lightglue, and an optimized UDIS fusion, along with the chroma balance algorithm, paves the way for further research into more sophisticated and robust image processing techniques. The authors believe their work contributes to advancing the field of image stitching and can be a basis for further improvements.

Summary updated

Abstract

Large field-of-view images are increasingly used in various environments today, and image stitching technology can make up for the limited field of view caused by hardware design. However, previous methods are constrained in various environments. In this paper, we propose a method that combines the powerful feature extraction capabilities of the Superpoint algorithm and the exact feature matching capabilities of the Lightglue algorithm with the image fusion algorithm of Unsupervised Deep Image Stitching (UDIS). Our proposed method effectively improves the situation where the linear structure is distorted and the resolution is low in the stitching results of the UDIS algorithm. On this basis, we make up for the shortcomings of the UDIS fusion algorithm. For stitching fractures of UDIS in some complex situations, we optimize the loss function of UDIS. We use a second-order differential Laplacian operator to replace the difference in the horizontal and vertical directions to emphasize the continuity of the structural edges during training. Combined with the above improvements, the Super Unsupervised Deep Image Stitching (SuperUDIS) algorithm is finally formed. SuperUDIS has better performance in both qualitative and quantitative evaluations compared to the UDIS algorithm, with the PSNR index increasing by 0.5 on average and the SSIM index increasing by 0.02 on average. Moreover, the proposed method is more robust in complex environments with large color differences or multi-linear structures.
BibTeX
@inproceedings{wu2024improved,
  author = {Haoze Wu and Chun Bao and Qun Hao and Jie Cao and Li Zhang},
  title = {{Improved Unsupervised Stitching Algorithm for Multiple Environments SuperUDIS}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24165352},
}
Figure