A self-adaptive motion estimation algorithm for superresolution reconstruction of multiframe SPOT panchromatic images

Bing Tan, Qing Xu, Jonathan Li, Yan ZhangCanadian 2007 · DOI

TL;DR

This paper introduces a novel self-adaptive motion estimation algorithm designed to improve the resolution of satellite imagery by combining feature point extraction, image matching, and wavelet-based superresolution. The method effectively handles local geometric distortions in satellite images, leading to significantly more accurate registration and higher-quality super-resolved images compared to traditional approaches.

What Problem Does This Paper Address?

High-resolution satellite imagery is crucial for applications like land use and land cover mapping, especially in urban and regional planning. While commercial high-resolution sensors exist, they are costly for large-area coverage. Existing lower-resolution satellite data (like Landsat or SPOT) offer wider coverage but lack the necessary detail. Super-resolution techniques offer a solution by reconstructing a high-resolution image from a sequence of lower-resolution images. However, a major challenge for satellite imagery is accurately estimating the subpixel motion between frames, which is complicated by local geometric distortions inherent in satellite imaging. Existing motion estimation methods often struggle with these distortions, leading to inaccurate registration and suboptimal super-resolution results.

Methods

The core of this paper is a self-adaptive motion estimation algorithm. This algorithm addresses the challenge of local geometric distortions by:

  • Feature Point Extraction: It utilizes a high-accuracy feature point extraction operator (like the Förstner operator) to identify key points in the reference image.
  • Least-Squares Matching (LSM): These feature points are then matched to corresponding tie points in other frames using LSM, aiming for subpixel accuracy.
  • Delaunay Triangulation: The image is initially divided into triangular regions using Delaunay triangulation based on these initial feature points.
  • Piecewise Affine Transform: Motion estimation is performed within each triangular region using a piecewise affine transform model, which is more robust to local distortions than a global model.
  • Self-Adaptive Refinement: The key innovation is the "self-adaptive" nature. The algorithm iteratively refines the regions. If the motion parameters between adjacent regions differ significantly (exceeding a threshold), more feature points are extracted within that region, splitting it into smaller subregions for more precise local motion estimation. Conversely, if parameters are similar, regions are merged. This process continues until the motion parameters across neighboring regions are consistent, ensuring the local distortion is well-captured.
  • Wavelet Superresolution: Once accurate motion parameters are estimated, the paper employs a wavelet-based superresolution technique. This method reconstructs the high-resolution image by estimating wavelet coefficients from the registered low-resolution frames.
  • HR Image Denoising and Deblurring: As a post-processing step, the paper discusses denoising and deblurring techniques (specifically mentioning Weiner filtering) to further enhance the quality of the reconstructed high-resolution image, addressing sensor blurring and noise.

Datasets

The paper evaluates its algorithm using three datasets:

  1. Simulated Dataset 1: Two subsections of an orthorectified SPOT-4 Pan image paired with a Digital Elevation Model (DEM). This dataset is designed to mimic the strict distortion geometry of linear pushbroom imagery.
  2. Simulated Dataset 2: An original high-resolution image that was blurred, down-sampled by a factor of 2, and then shifted randomly to create five low-resolution frames. This simulates motion and degradation.
  3. Real Dataset (SPOT-4 Pan Images): Seven SPOT-4 Pan images with varying acquisition dates and viewing angles.

Key Insights and Ideas

  • Local Distortion is Key: The paper strongly emphasizes that satellite imagery's geometric distortions are often non-global. A self-adaptive approach that refines motion estimation based on local consistency is crucial for accurate registration.
  • Iterative Refinement for Robustness: The iterative merging and splitting of triangular regions based on motion parameter consistency is a clever way to automatically adapt the motion estimation to varying levels of local distortion without manual intervention.
  • Wavelets for Superresolution: The use of wavelet-based superresolution is highlighted as an effective method for reconstructing high-resolution images from registered, albeit degraded, low-resolution frames.
  • Motion Estimation as a Precursor: The paper clearly positions accurate motion estimation as a critical prerequisite for successful wavelet superresolution, especially for satellite imagery.

Key Results

  • Subpixel Accuracy: The proposed self-adaptive motion estimation algorithm achieved a maximum error of 0.6 pixels and an average error of 0.2 pixels on simulated data, significantly outperforming a gradient-based method (max error 4.7, average 3.2 pixels).
  • Improved Super-Resolution Quality: Visual inspection of reconstructed SPOT-4 Pan images showed a higher level of detail compared to the original low-resolution frames.
  • Quantitative Positioning Accuracy: Object positioning experiments on a reconstructed SPOT-4 image showed better accuracy (lower positioning error) than the original LR image, implying higher resolution. For example, the average positioning error for the reconstructed HR image was 6.467 m, compared to 32.914 m for the original LR image.
  • MSE Improvement: For simulated dataset 2, the reconstructed superresolution image had a Mean Squared Error (MSE) of 9.87, which was considerably lower than the MSE of a simple averaging of registered frames (23.97) and the simulated LR image (32.99).

Limitations and Future Work

  • Computational Complexity: The authors acknowledge that as image dimensions increase, the computational complexity of the wavelet superresolution algorithm can become a bottleneck, especially for very large satellite images (e.g., 6000x6000 pixels). Improving computation efficiency is a key future challenge.
  • Temporal Differences: Significant differences in images acquired over long time intervals (due to changes in the scene or acquisition conditions) might pose problems for the proposed algorithm.
  • Generalizability: While the method is demonstrated on SPOT-4 Pan images, its performance on other sensor types or with different distortion characteristics could be further explored.
  • Denoising/Deblurring Integration: While denoising and deblurring are discussed as post-processing steps, a more integrated approach within the superresolution framework could potentially yield even better results.

Summary updated

Abstract

High spatial resolution of imaging sensors onboard Earth observation satellites is often critical for land use and land cover mapping. In this paper, we propose a self-adaptive motion estimation algorithm that combines the consistency characteristics of image distortion, high accuracy feature point extraction, and image matching techniques to achieve subpixel accuracy of motion estimation. Using the wavelet superresolution and the modulation transfer function (MTF), we implemented the superresolution reconstruction from multiframe Satellite pour l’Observation de la Terre (SPOT-4) panchromatic images under a framework of interpolation–restoration. The results show that this algorithm is promising for reconstruction of a higher resolution frame.
BibTeX
@inproceedings{tan2007self,
  author = {Bing Tan and Qing Xu and Jonathan Li and Yan Zhang},
  title = {{A self-adaptive motion estimation algorithm for superresolution reconstruction of multiframe SPOT panchromatic images}},
  booktitle = {Canadian},
  year = {2007},
  doi = {10.5589/m07-052},
}
Figure