Abstract
Unsupervised deep image stitching often struggles with scenes containing significant parallax and low-texture regions. We present UDIS++, a robust framework that addresses these challenges through a two-stage pipeline. By combining global homography with local Thin-Plate Spline (TPS) warping and a novel seam-driven composition network, UDIS++ achieves sharp, artifact-free results. Our method outperforms existing state-of-the-art approaches on the UDIS-D dataset, offering superior alignment accuracy and computational efficiency.
Introduction: The Parallax and Texture Challenge
Image stitching—the process of combining multiple images into a single wide-angle panorama—is a fundamental task in computer vision. However, traditional methods often fail in two specific scenarios: large parallax and low texture.
Parallax occurs when objects at different depths appear to shift relative to one another as the camera moves, violating the "flat-plane" assumption of standard homography and causing "ghosting" artifacts. Low-texture environments, such as smooth walls or dark hallways, lack the distinct features (like SIFT keypoints) that traditional algorithms rely on for tracking. While deep learning has improved semantic understanding, previous unsupervised models often relied on global warps that couldn't handle local parallax or used reconstruction techniques that resulted in unnatural blurring.
Methodology: UDIS++ Pipeline
1. Global-to-Local Warping
UDIS++ employs a two-stage warping strategy to ensure both general alignment and local flexibility:
- Global Stage: A regression network estimates a 4-point homography to establish a coarse alignment between the reference and target images.
- Local Stage (TPS): To handle parallax, the model predicts residual motions for a grid of control points using Thin-Plate Splines (TPS). Unlike traditional mesh-based warps, TPS allows for smooth, non-linear deformations. To prevent geometric distortion, the authors introduce "intra-grid" and "inter-grid" constraints that act as virtual springs, maintaining the structural integrity of the warped image.
2. Seam-Driven Composition
After warping, images rarely align perfectly due to parallax. Instead of using reconstruction-based blending—which often causes blur—UDIS++ utilizes a seam-driven composition network. A UNet-style architecture generates composition masks that learn to "cut" between images at natural boundaries. By applying boundary and smoothness constraints, the network hides parallax artifacts while preserving the original sharpness and texture of the input images.
3. Iterative Warp Adaption
To enhance generalization across different datasets, UDIS++ includes an iterative strategy that fine-tunes the warp on-the-fly for specific image pairs. This process significantly improves alignment in a fraction of a second, making the model robust to unseen environments.
Key Insights and Results
A major engineering contribution of this work is the preference for TPS over traditional meshes. Mesh-based multi-homography methods are difficult to accelerate on GPUs because pixel-to-grid mapping is computationally expensive. In contrast, TPS treats the image as a continuous function, enabling 15-20x faster processing on high-resolution images.
Quantitatively, UDIS++ achieved a PSNR of 30.19 and SSIM of 0.933 on the UDIS-D dataset. As shown in the figure above, UDIS++ successfully aligns images in "impossible" cases where traditional SIFT-based methods fail to find enough features and where previous deep learning models like LPC produce significant ghosting or blurring.
Conclusion
UDIS++ represents a significant advancement in unsupervised image stitching. By moving from reconstruction-based blending to seam-driven composition and utilizing GPU-friendly TPS warps, the framework provides a fast and effective solution for complex real-world scenes. While the iterative adaptation adds slight overhead, the overall gains in visual quality and robustness make it a powerful tool for generating high-fidelity panoramas.
Summary updated