Robust Alignment of Multi-Exposed Images with Saturated Regions

Jun Jiang, Zhengguo Li, Shoulie Xie, Shiqian Wu, Liangcai Zeng

Abstract

It is challenging to align multi-exposed images due to large illumination variations, especially in presence of saturated regions. In this paper, a novel image alignment algorithm is proposed to cope with the multi-exposed images with saturated regions. Specifically, the multi-exposed images are first normalized by using intensity mapping functions (IMFs) in consideration of saturated pixels. Then, the normalized images are coded by using the local binary pattern (LBP). Finally, the coded images are aligned by formulating an optimization problem by using a differentiable Hamming distance. Experimental results show that the proposed algorithm outperforms state-of-the-art alignment methods for multi-exposed images in terms of alignment accuracy and robustness to exposure values.

Introduction

Aligning images with significantly different exposure levels is a fundamental challenge in computer vision. Applications such as High Dynamic Range (HDR) imaging, panorama stitching, and Simultaneous Localization and Mapping (SLAM) frequently require registering frames where features may disappear due to over-exposure (saturation) or under-exposure (shadows). Standard alignment techniques often rely on the "brightness constancy assumption," which posits that pixel intensities remain relatively stable across frames. This assumption fails spectacularly in multi-exposure scenarios.

A challenging real-world multi-exposure sequence ('Inscription') featuring severe saturation and deep shadows.

Furthermore, saturated regions—areas of pure white or black—introduce significant noise into the alignment process. A feature visible in a low-exposure shot might be completely washed out in a high-exposure counterpart. While methods like Median Threshold Bitmaps (MTB) offer some robustness, they often lack sub-pixel accuracy. Conversely, modern learning-based descriptors like SuperPoint frequently fail when exposure differences become extreme, often finding zero matching pairs at intervals of 7EV to 8EV.

Methodology

The proposed framework addresses these issues through a two-stage pipeline: bi-directional normalization and optimization via a differentiable Hamming distance.

Bi-directional Normalization and LBP Coding

Instead of a simple unidirectional brightness mapping, the authors employ Intensity Mapping Functions (IMFs) to synchronize the images. They introduce specific constants to handle under- and over-exposed pixels, ensuring that if a region is saturated in one image, the corresponding region in the other is treated consistently. This "saturation synchronization" prevents the cost function from being misled by features that exist in only one frame of the pair. Following normalization, the images are coded using Local Binary Patterns (LBP), which are naturally invariant to monotonic lighting changes.

Overview of the proposed alignment pipeline: (a) input multi-exposed pair, (b) intensity mapping functions, (c) normalized images, and (d) LBP coded images.

Differentiable Hamming Distance

The primary innovation lies in how these LBP codes are optimized. Typically, the Hamming distance used to compare binary strings is discrete and non-differentiable, precluding gradient-based optimization. Moreover, previous methods often fell into the "decimal trap"—converting 8-bit LBP strings to decimal numbers (0-255) and using Euclidean distance. This is flawed because a single bit flip can change a decimal value by 1 or by 128, which does not accurately reflect the similarity of the descriptors.

The authors overcome this by proving that for binary values $\{0, 1\}$, the squared Euclidean distance $(a-b)^2$ is mathematically equivalent to the XOR operation ($a \oplus b$). By treating LBP bits as individual dimensions in a vector, the alignment problem is transformed into a continuous quadratic optimization problem. This allows for rapid, sub-pixel accurate registration using Taylor expansion and linear regression.

Experimental Evaluation

The method was evaluated on a diverse set of data, including benchmark sequences with synthetic motion, 37 multi-exposure sequences from Cai’s dataset, and 35 real-world sequences captured with professional DSLRs. These datasets include notorious test cases like "BigTree" and "Pillar," characterized by extreme sun glare and deep shadows.

Key Results

  • Accuracy: On benchmark datasets, the proposed method achieved a mean rotation error of 0.6° and translation errors between 1.8 and 3.8 pixels, significantly outperforming MTB, SIFT, and several deep learning models.
  • Robustness: The algorithm remained stable at exposure intervals as high as 8EV. In contrast, traditional feature-based methods like SIFT often fail entirely at these extremes.
  • Efficiency: By formulating the optimization as a linear regression, the method is approximately 2.5 times faster than hybrid approaches like IMF+SIFT while maintaining superior accuracy.

Conclusion

This research bridges the gap between the robustness of binary descriptors and the efficiency of continuous optimization. By synchronizing saturated regions and introducing a differentiable Hamming distance, the proposed algorithm provides a fast and reliable solution for multi-exposed image alignment. Future work will explore integrating this framework into autonomous SLAM systems and tele-operation platforms where lighting conditions are highly unpredictable.

Summary updated

Abstract

It is challenging to align multi-exposed images due to large illumination variations, especially in presence of saturated regions. In this paper, a novel image alignment algorithm is proposed to cope with the multi-exposed images with saturated regions. Specifically, the multi-exposed images are first normalized by using intensity mapping functions (IMFs) in consideration of saturated pixels. Then, the normalized images are coded by using the local binary pattern (LBP). Finally, the coded images are aligned by formulating an optimization problem by using a differentiable Hamming distance. Experimental results show that the proposed algorithm outperforms state-of-the-art alignment methods for multi-exposed images in terms of alignment accuracy and robustness to exposure values.
BibTeX
@inproceedings{jiang2020robust,
  author = {Jun Jiang and Zhengguo Li and Shoulie Xie and Shiqian Wu and Liangcai Zeng},
  title = {{Robust Alignment of Multi-Exposed Images with Saturated Regions}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=kKbYv-vAk8z},
}
Figure