EC-WAMI: Event Camera-Based Pose Optimization in Remote Sensing and Wide-Area Motion Imagery

Isaac Nkrumah, Maryam Moshrefizadeh, Omar Tahri, Erik Blasch, Kannappan Palaniappan, Hadi AliakbarpourSensors 2024 · DOI

TL;DR

This paper introduces EC-WAMI, the first system to leverage event cameras for Structure-from-Motion (SfM) in Wide-Area Motion Imagery (WAMI) and Remote Sensing (RS). By simulating event data from traditional aerial imagery, they demonstrate that event cameras can achieve accurate camera pose optimization and 3D reconstruction, even under challenging lighting conditions, and significantly outperform traditional methods in speed.

What Problem Does This Paper Address?

Traditional aerial imaging and remote sensing often struggle with challenging lighting conditions like glare, uneven illumination, and low light. These issues can lead to motion blur and reduced temporal resolution, making it difficult for standard Structure-from-Motion (SfM) techniques to accurately estimate camera poses and reconstruct 3D scenes. While SfM is crucial for tasks like situational awareness and object tracking in WAMI, these limitations hinder its effectiveness. The paper aims to address this by exploring the potential of event cameras (ECs), which are designed to overcome these very challenges.

Methods

The core of this work is the development of the EC-WAMI pipeline. Since real-world event camera WAMI datasets are scarce, the authors simulate event data from existing RGB WAMI datasets using the Video-to-Event (V2E) algorithm. These simulated event streams are then converted back into frames using the Event-to-Frame (E2F) method (specifically E2VID). These reconstructed frames, along with their associated metadata, are fed into two state-of-the-art SfM pipelines: COLMAP and BA4S.

  • Event Simulation: V2E converts RGB video frames into event streams, mimicking how an event camera would perceive the scene.
  • Frame Reconstruction: E2VID reconstructs grayscale frames from these event streams.
  • SfM Pipelines:
    • COLMAP: A traditional, comprehensive SfM pipeline that performs exhaustive feature matching and bundle adjustment.
  • BA4S (Bundle Adjustment for Sequential Imagery): A more efficient SfM pipeline designed for sequential data, which directly uses metadata and a robust error function to mitigate outliers, making it suitable for computationally intensive applications like WAMI.
  • Feature Extraction: The paper evaluates various feature detectors and descriptors, including traditional ones like SIFT, ORB, AKAZE, DCTF, SURF, and a deep learning-based descriptor called LIGHTGLUE.

Datasets

The authors utilize two datasets for their experiments:

  • ABQ-215: A dataset of real RGB aerial images from Albuquerque's downtown area. The original high-resolution images were downsampled for computational efficiency.
  • DIRSIG-RIT: A synthetic RGB imagery dataset from Digital Imaging and Remote Sensing Image Generation at the Rochester Institute of Technology. This dataset includes ground-truth metadata.

The paper also introduces the EC-WAMI dataset (made publicly available), which consists of simulated event data derived from these RGB datasets.

Key Insights and Ideas

  • Event Cameras for WAMI/RS is a Novel Frontier: This paper pioneers the use of event cameras for SfM in WAMI and RS, opening up new possibilities for robust 3D reconstruction in challenging environments.
  • Event Data Can Enable Accurate Pose Recovery Even at Low Frame Rates: Despite the simulated 5 fps frame rate, the event-based approach achieved accurate camera pose recovery, demonstrating the inherent robustness of event data.
  • BA4S Outperforms COLMAP in Speed for WAMI: For WAMI applications, BA4S offers a significant speed advantage (over 200x faster) compared to COLMAP while maintaining comparable accuracy, making it more practical for real-time or large-scale processing.
  • Deep Learning Features Shine: The LIGHTGLUE descriptor consistently outperformed traditional handcrafted features across various metrics, highlighting the power of deep learning for robust feature matching in event-based vision.
  • Event Cameras Excel Where Traditional Cameras Fail: The paper demonstrates that traditional RGB cameras struggle significantly in challenging illumination and motion scenarios, whereas event cameras, even when simulated, provide a viable solution.

Key Results

  • Comparable Accuracy, Superior Speed: BA4S and COLMAP showed comparable accuracy in camera pose estimation, but BA4S was over 200 times faster.
  • LIGHTGLUE Dominates Feature Performance: The LIGHTGLUE learned descriptor achieved the best results across almost all positional and angular error metrics on both datasets.
  • Handcrafted Features Show Promise: AKAZE and DCTF performed competitively among handcrafted features, with DCTF showing robustness in challenging conditions.
  • Robust 3D Reconstruction: Both sparse and dense 3D reconstructions were successfully generated, with Gaussian splatting producing high-quality results.
  • Failure of Traditional Cameras: In simulated challenging conditions (motion blur, low light), traditional RGB cameras failed to recover camera poses, underscoring the limitations of current methods.

Limitations and Future Work

Limitations:

  • Simulated Data: The current work relies on simulated event data due to the scarcity of real-world event camera WAMI datasets. This simulation might not perfectly capture all nuances of real event camera data.
  • Frame Reconstruction Dependency: The current pipeline relies on frame reconstruction, which can introduce inefficiencies and potential artifacts.

Future Work:

  • Real Event Camera Data: Collecting and utilizing real airborne event camera datasets for WAMI and RS.
  • Direct Event Stream Processing: Developing algorithms that process event streams directly, eliminating the need for intermediate frame reconstruction.
  • Computational Efficiency: Further improving computational efficiency by integrating event data directly into the optimization engine.
  • Deployment Options: Exploring deployment strategies such as containerized approaches for practical applications.

Summary updated

Abstract

In this paper, we present <i>EC-WAMI</i>, the first successful application of neuromorphic <i>event cameras (ECs)</i> for Wide-Area Motion Imagery (WAMI) and Remote Sensing (RS), showcasing their potential for advancing Structure-from-Motion (SfM) and 3D reconstruction across diverse imaging scenarios. ECs, which detect asynchronous pixel-level <i>brightness changes</i>, offer key advantages over traditional frame-based sensors such as high temporal resolution, low power consumption, and resilience to dynamic lighting. These capabilities allow ECs to overcome challenges such as glare, uneven lighting, and low-light conditions that are common in aerial imaging and remote sensing, while also extending UAV flight endurance. To evaluate the effectiveness of ECs in WAMI, we simulate event data from RGB WAMI imagery and integrate them into SfM pipelines for camera pose optimization and 3D point cloud generation. Using two state-of-the-art SfM methods, namely, COLMAP and Bundle Adjustment for Sequential Imagery (BA4S), we show that although ECs do not capture scene content like traditional cameras, their spike-based events, which only measure <i>illumination changes</i>, allow for accurate camera pose recovery in WAMI scenarios even in low-framerate(5 fps) simulations. Our results indicate that while BA4S and COLMAP provide comparable accuracy, BA4S significantly outperforms COLMAP in terms of speed. Moreover, we evaluate different feature extraction methods, showing that the deep learning-based LIGHTGLUE descriptor consistently outperforms traditional handcrafted descriptors by providing improved reliability and accuracy of event-based SfM. These results highlight the broader potential of ECs in remote sensing, aerial imaging, and 3D reconstruction beyond conventional WAMI applications. Our dataset will be made available for public use.
BibTeX
@inproceedings{nkrumah2024ec,
  author = {Isaac Nkrumah and Maryam Moshrefizadeh and Omar Tahri and Erik Blasch and Kannappan Palaniappan and Hadi Aliakbarpour},
  title = {{EC-WAMI: Event Camera-Based Pose Optimization in Remote Sensing and Wide-Area Motion Imagery}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s24237493},
}
Figure