Feature Matching Combining Radiometric and Geometric Characteristics of Images, Applied to Oblique- and Nadir-Looking Visible and TIR Sensors of UAV Imagery

Hyo-Seon Jang, Sangkyun Kim, Suhong Yoo, Soohee Han, Hong-Gyoo SohnSensors 2021 · OpenReview

TL;DR

This paper introduces a novel image matching method (PCH-IMAS) that effectively combines thermal infrared (TIR) and visible light imagery from UAVs, even when captured from different angles (nadir vs. oblique). This approach overcomes significant challenges in aligning these diverse datasets, enabling more robust feature extraction for applications like urban monitoring and disaster management.

What Problem Does This Paper Address?

Accurately identifying and mapping features on the ground is crucial for many applications, from urban planning to disaster response. Thermal infrared (TIR) imagery offers valuable information, especially at night or for detecting heat signatures, which visible light imagery cannot provide. However, TIR images often have lower resolution and different radiometric properties compared to visible images. Furthermore, using Unmanned Aerial Vehicles (UAVs) to capture data from various angles (nadir-looking straight down, and oblique-looking at an angle) introduces significant geometric distortions. Matching features between images with different wavelengths (visible vs. TIR) and different viewing geometries (nadir vs. oblique) is a notoriously difficult problem, leading to poor correspondence and unreliable results with existing methods. This paper aims to bridge this gap by developing a robust method for matching such challenging image pairs.

Methods

The proposed method, termed PCH-IMAS (Phase Congruency with Histogram – Image Matching by Affine Simulation), is a three-stage process designed to handle the radiometric and geometric differences:

  1. Combined Image Generation (Phase Congruency): This stage uses phase congruency (PC) in the frequency domain to extract edge information from both visible and TIR images. PC is known for its robustness to illumination changes. The maximum moment of PC is calculated and combined with the original image to create an "edge-enhanced" image. This process aims to create a common representation that highlights similar features across different wavelengths.
  2. Histogram Matching: To address the radiometric differences between visible and TIR images (e.g., variations due to sun illumination), histogram matching is applied. The histogram of the combined visible image is adjusted to match the distribution of the combined TIR image. This step aims to normalize the pixel intensity distributions, making the images more comparable.
  3. Image Matching by Affine Simulation (IMAS): Finally, to handle the geometric disparities introduced by nadir and oblique views, the IMAS technique is employed. IMAS is designed to simulate images from various angles and create "hyper-descriptors" for robust feature matching, even with significant geometric transformations like affine transformations (scaling, rotation, shear, translation).

Datasets

The study utilized custom-collected UAV imagery from Buho-ri, South Korea. This dataset includes:

  • Visible nadir-looking images
  • Visible oblique images
  • TIR nadir-looking images
  • TIR oblique images

These images were captured over both building-dense and non-building (rural/field) areas. The paper does not mention using pre-existing benchmark datasets for training or evaluation, emphasizing the custom nature of their data acquisition to address the specific problem of matching diverse geometries and wavelengths.

Key Insights and Ideas

  • Synergy of Wavelengths and Geometries: The core idea is that combining information from both visible and TIR sensors, and accounting for different viewing angles, unlocks a powerful synergy for detailed surface analysis that neither sensor nor geometry can achieve alone.
  • Phase Congruency for Cross-Wavelength Edge Detection: Using phase congruency, which is less sensitive to illumination, proves effective in extracting edge information that can be meaningfully compared between visible and TIR images, despite their inherent differences.
  • Histogram Matching as a Radiometric Normalizer: Adjusting the histogram of one image to match another is a crucial step in bridging the radiometric gap, making features more comparable before geometric matching.
  • IMAS for Geometric Robustness: The IMAS technique is highlighted as a key enabler for handling the significant geometric distortions introduced by oblique imaging, a common challenge with UAV data.
  • Bare Ground as a Robust Feature: For non-building areas, features on bare ground (like furrows or changes in texture) were found to be more robust to geometric and radiometric variations than features on flat, uniform surfaces.
  • Shadows as a Matching Challenge: Shadows, especially in oblique imagery, can significantly hinder matching by creating dark, undifferentiated areas. The paper suggests pre-computing shadow directions to mitigate this.

Key Results

The proposed PCH-IMAS method demonstrated superior performance compared to established methods like SIFT, SURF, SAR-SIFT, and ASIFT.

  • Matching Success: The proposed method achieved successful matching in all tested cases (building/non-building, visible nadir vs. TIR oblique, and visible oblique vs. TIR nadir-looking), whereas the other methods failed to produce matches.
  • Accuracy: The Root Mean Square Error (RMSE) for the proposed method was approximately 20 pixels, significantly lower than the "not matched" results of other methods or accuracies in the hundreds of pixels for comparable methods.
  • Inlier Count: The number of inliers (correctly matched points) for the proposed method was substantial (e.g., 24 for building, visible nadir vs. TIR oblique; 32 for building, visible oblique vs. TIR nadir-looking), indicating a high degree of reliable correspondence. Building-type images generally yielded more inliers due to richer feature variations.

Limitations and Future Work

  • Dataset Specificity: The study relied on a custom-collected dataset. While this allowed for tailored data acquisition, the generalizability to other geographic locations or sensor types might require further validation.
  • Shadow Mitigation: While the paper identifies shadows as a challenge and suggests pre-computation, more advanced automated shadow detection and removal techniques could further enhance robustness.
  • Computational Cost: While not explicitly detailed, complex multi-stage methods like this can be computationally intensive. Future work could focus on optimizing the algorithms for faster processing.
  • Generalization to More Complex Scenarios: While the method performed well on building and non-building types, extending it to even more complex urban environments with diverse object types and extreme geometric variations could be a future direction.
  • Automated Parameter Tuning: The paper mentions optimal selection of experimental locations and data acquisition, implying some manual tuning. Developing more automated parameter selection for different environments could be beneficial.

Summary updated

Abstract

A large amount of information needs to be identified and produced during the process of promoting projects of interest. Thermal infrared (TIR) images are extensively used because they can provide information that cannot be extracted from visible images. In particular, TIR oblique images facilitate the acquisition of information of a building’s facade that is challenging to obtain from a nadir image. When a TIR oblique image and the 3D information acquired from conventional visible nadir imagery are combined, a great synergy for identifying surface information can be created. However, it is an onerous task to match common points in the images. In this study, a robust matching method of image pairs combined with different wavelengths and geometries (i.e., visible nadir-looking vs. TIR oblique, and visible oblique vs. TIR nadir-looking) is proposed. Three main processes of phase congruency, histogram matching, and Image Matching by Affine Simulation (IMAS) were adjusted to accommodate the radiometric and geometric differences of matched image pairs. The method was applied to Unmanned Aerial Vehicle (UAV) images of building and non-building areas. The results were compared with frequently used matching techniques, such as scale-invariant feature transform (SIFT), speeded-up robust features (SURF), synthetic aperture radar–SIFT (SAR–SIFT), and Affine SIFT (ASIFT). The method outperforms other matching methods in root mean square error (RMSE) and matching performance (matched and not matched). The proposed method is believed to be a reliable solution for pinpointing surface information through image matching with different geometries obtained via TIR and visible sensors.
BibTeX
@inproceedings{jang2021feature,
  author = {Hyo-Seon Jang and Sangkyun Kim and Suhong Yoo and Soohee Han and Hong-Gyoo Sohn},
  title = {{Feature Matching Combining Radiometric and Geometric Characteristics of Images, Applied to Oblique- and Nadir-Looking Visible and TIR Sensors of UAV Imagery}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=aBMSFxATp7},
}
Figure