Abstract
We investigate projective estimation under model inadequacies, i.e., when the underpinning assumptions of the projective model are not fully satisfied by the data. We focus on the task of image stitching which is customarily solved by estimating a projective warp - a model that is justified when the scene is planar or when the views differ purely by rotation. Such conditions are easily violated in practice, and this yields stitching results with ghosting artefacts that necessitate the usage of deghosting algorithms. To this end we propose as-projective-as-possible warps, i.e., warps that aim to be globally projective, yet allow local non-projective deviations to account for violations to the assumed imaging conditions. Based on a novel estimation technique called Moving Direct Linear Transformation (Moving DLT), our method seamlessly bridges image regions that are inconsistent with the projective model. The result is highly accurate image stitching, with significantly reduced ghosting effects, thus lowering the dependency on post hoc deghosting.
Introduction: The Limits of Global Homographies
In computer vision, the standard approach to stitching two images involves estimating a $3 \times 3$ homography matrix. This model assumes either a perfectly planar scene or a camera rotating around its optical center. However, real-world conditions—particularly handheld photography—frequently violate these assumptions. When a camera translates, objects at varying depths exhibit parallax, shifting at different relative speeds and causing misalignments in the final stitch.
Traditionally, these errors are addressed post-warping using "deghosting" algorithms like seam cutting or sophisticated blending. This paper proposes a different approach: fixing the warp itself. The goal is to create a warp flexible enough to accommodate local misalignments while remaining rigid enough to preserve a realistic global structure.
Methods: Moving Direct Linear Transformation
The core innovation of this work is Moving Direct Linear Transformation (Moving DLT). This technique allows for a spatially varying warp that adapts to local image features.
- Local Weighting: Rather than calculating a single global homography, the algorithm computes a unique homography for every point (or grid cell) in the image. It leverages feature matches (e.g., SIFT points) and assigns higher weights to matches physically closer to the point being warped.
- As-Projective-As-Possible (APAP): To prevent unrealistic distortions in regions lacking feature matches (extrapolation), the authors introduce a regularization term ($\gamma$). This term constrains local warps to remain as close as possible to a global projective model.
- Computational Efficiency: To avoid the prohibitive cost of a full Singular Value Decomposition (SVD) for every pixel, the image is partitioned into a grid. A "rank-one update" trick is used to update the SVD from a base solution, making the process practical for real-world applications.
Datasets and Evaluation
The APAP method was evaluated on several challenging real-world datasets characterized by significant parallax and complex geometry:
- Railtracks: A scene with deep perspective and parallel lines, highly prone to ghosting.
- Temple: Featuring complex architectural details.
- Carpark, Chess/Girl, and Rooftops: Testing various levels of texture and depth.
Quantitative analysis using synthetic 3D point clouds demonstrated that APAP consistently maintains lower Root Mean Squared Error (RMSE) as camera translation increases compared to standard methods.
Key Insights and Results
- The "Moving" Concept: By adapting "Moving Least Squares" logic to the projective domain, the authors created a "seamless bridge" between regions that do not fit a global model.
- Interpolation vs. Extrapolation: Projective regularization ensures that even in data-sparse regions, the stitch remains geometrically plausible, avoiding the failures often seen with purely affine warps.
- Reduced Post-Processing: By minimizing alignment error during the warping stage, the need for complex deghosting is significantly reduced. Simple pixel averaging often suffices to produce clean results.
- Performance: APAP achieves superior accuracy and visual quality while remaining computationally efficient, taking less than a minute for medium-sized images—a significant improvement over previous state-of-the-art methods like Smoothly Varying Affine (SVA).
Conclusion
As-Projective-As-Possible (APAP) warping provides a robust solution for image stitching under non-ideal conditions. While it is designed to refine rather than replace the projective model, it significantly reduces ghosting artifacts caused by parallax. Future work includes extending the method to simultaneous multi-image alignment for full panoramas and further optimizing the Moving DLT calculation for real-time performance.
Summary updated