Learning Thin-Plate Spline Motion and Seamless Composition for Parallax-Tolerant Unsupervised Deep Image Stitching

Lang Nie, Chunyu Lin, Kang Liao, Shuaicheng Liu, Yao Zhao

Abstract

Unsupervised deep image stitching often struggles with scenes containing significant parallax and low-texture regions. We present UDIS++, a robust framework that addresses these challenges through a two-stage pipeline. By combining global homography with local Thin-Plate Spline (TPS) warping and a novel seam-driven composition network, UDIS++ achieves sharp, artifact-free results. Our method outperforms existing state-of-the-art approaches on the UDIS-D dataset, offering superior alignment accuracy and computational efficiency.

Introduction: The Parallax and Texture Challenge

Image stitching—the process of combining multiple images into a single wide-angle panorama—is a fundamental task in computer vision. However, traditional methods often fail in two specific scenarios: large parallax and low texture.

Parallax occurs when objects at different depths appear to shift relative to one another as the camera moves, violating the "flat-plane" assumption of standard homography and causing "ghosting" artifacts. Low-texture environments, such as smooth walls or dark hallways, lack the distinct features (like SIFT keypoints) that traditional algorithms rely on for tracking. While deep learning has improved semantic understanding, previous unsupervised models often relied on global warps that couldn't handle local parallax or used reconstruction techniques that resulted in unnatural blurring.

Methodology: UDIS++ Pipeline

1. Global-to-Local Warping

UDIS++ employs a two-stage warping strategy to ensure both general alignment and local flexibility:

  • Global Stage: A regression network estimates a 4-point homography to establish a coarse alignment between the reference and target images.
  • Local Stage (TPS): To handle parallax, the model predicts residual motions for a grid of control points using Thin-Plate Splines (TPS). Unlike traditional mesh-based warps, TPS allows for smooth, non-linear deformations. To prevent geometric distortion, the authors introduce "intra-grid" and "inter-grid" constraints that act as virtual springs, maintaining the structural integrity of the warped image.

2. Seam-Driven Composition

After warping, images rarely align perfectly due to parallax. Instead of using reconstruction-based blending—which often causes blur—UDIS++ utilizes a seam-driven composition network. A UNet-style architecture generates composition masks that learn to "cut" between images at natural boundaries. By applying boundary and smoothness constraints, the network hides parallax artifacts while preserving the original sharpness and texture of the input images.

3. Iterative Warp Adaption

To enhance generalization across different datasets, UDIS++ includes an iterative strategy that fine-tunes the warp on-the-fly for specific image pairs. This process significantly improves alignment in a fraction of a second, making the model robust to unseen environments.

Key Insights and Results

A major engineering contribution of this work is the preference for TPS over traditional meshes. Mesh-based multi-homography methods are difficult to accelerate on GPUs because pixel-to-grid mapping is computationally expensive. In contrast, TPS treats the image as a continuous function, enabling 15-20x faster processing on high-resolution images.

Qualitative comparison of UDIS++ against LPC and UDIS on large parallax and low-texture cases.

Quantitatively, UDIS++ achieved a PSNR of 30.19 and SSIM of 0.933 on the UDIS-D dataset. As shown in the figure above, UDIS++ successfully aligns images in "impossible" cases where traditional SIFT-based methods fail to find enough features and where previous deep learning models like LPC produce significant ghosting or blurring.

Conclusion

UDIS++ represents a significant advancement in unsupervised image stitching. By moving from reconstruction-based blending to seam-driven composition and utilizing GPU-friendly TPS warps, the framework provides a fast and effective solution for complex real-world scenes. While the iterative adaptation adds slight overhead, the overall gains in visual quality and robustness make it a powerful tool for generating high-fidelity panoramas.

Summary updated

BibTeX
@inproceedings{nie2023learning,
  author = {Lang Nie and Chunyu Lin and Kang Liao and Shuaicheng Liu and Yao Zhao},
  title = {{Learning Thin-Plate Spline Motion and Seamless Composition for Parallax-Tolerant Unsupervised Deep Image Stitching}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00680},
}
Figure