Abstract
Multi-image alignment, bringing a group of images into common register, is a ubiquitous problem and the first step of many applications in a wide variety of domains. As a result, a great amount of effort is being invested in developing efficient multi-image alignment algorithms. Little has been done, however, to answer fundamental practical questions such as: what is the comparative performance of existing methods? is there still room for improvement? under which conditions should one technique be preferred over another? does adding more images or prior image information improve the registration results? In this work, we present a thorough analysis and evaluation of the main multi-image alignment methods which, combined with theoretical limits in multi-image alignment performance, allows us to organize them under a common framework and provide practical answers to these essential questions.
Introduction
Multi-image alignment is the "square one" for many computer vision tasks, including HDR imaging, super-resolution, and burst denoising. If alignment is inaccurate, downstream results suffer from ghosting or resolution loss. While numerous algorithms exist, researchers often lack a clear guide on selection criteria. This paper addresses these gaps by comparing existing methods against theoretical performance limits, specifically investigating the impact of image priors and the number of images on registration accuracy.
A Unified Mathematical Framework
The authors focus on 2D rigid translations, common in handheld photography. They unify several prominent methods under a single optimization framework:
- Maximum Likelihood Estimator (MLE): Minimizes the distance between observed images and their average.
- Maximum a Posteriori (MAP) & Bayesian MLE (BMLE): Incorporate a stationary Gaussian process as an image prior.
- Constrained Alignment (CA): Uses redundancy in pairwise shifts to find a globally consistent solution.
A key theoretical insight is that MLE, MAP, and BMLE optimize a nearly identical cost function. The primary difference is the inclusion of a Wiener filter, which acts as the image prior. Without this prior, Bayesian methods effectively revert to standard MLE.
Experimental Setup
The study utilizes two main data sources:
- Synthetic Data: $50 \times 50$ pixel patches from natural images, subjected to varying levels of additive Gaussian noise and different motion models (independent random shifts vs. drift trajectories).
- Real-World Data: Image bursts captured with a Sony $\alpha$-5100 camera at high ISO (20000) to simulate low-light, high-noise conditions.
Key Insights and Results
The SNR Threshold and the Power of Priors
There is a distinct Signal-to-Noise Ratio (SNR) "cliff" below which alignment becomes mathematically impossible. For standard MLE, this threshold is around -10dB. However, incorporating an image prior (MAP) can push this failure threshold back to -14dB.
As shown in the figure above, while all methods converge to the Cramér-Rao Bound (CRB) in high-SNR conditions, the MAP estimator provides a significant advantage in noisy environments. Notably, a MAP estimator using only 5 images can outperform an MLE estimator using 10 images, highlighting that "smarter" algorithms are often more efficient than simply increasing data quantity.
The Critical Role of Initialization
In low-SNR regimes, the optimization landscape becomes highly non-convex. Starting the optimization with random shifts often leads to failure.
The figure above illustrates that for MLE, proper initialization (e.g., via simple pairwise correlation) is essential for the joint optimization methods to converge to the global optimum as SNR decreases.
Conclusion
This work demonstrates that while most multi-image alignment methods share a common functional core, the inclusion of image priors is the single most effective way to achieve accurate alignment in extremely noisy environments. While the current framework focuses on 2D rigid translations, it provides a robust foundation for future extensions into non-rigid transformations or deep-learning-based priors.
Summary updated