Learned Event-Based Visual Perception for Improved Space Object Detection

Nikolaus Salvatore, Justin FletcherWACV 2022

TL;DR

This paper introduces a novel hybrid deep learning architecture that combines traditional image features with event-based sensor data to improve the detection of dim, artificial satellites in Earth orbit. By leveraging the high temporal resolution and dynamic range of event cameras, the proposed method achieves better performance, especially in challenging scenarios with significant background noise.

What Problem Does This Paper Address?

The increasing number of artificial satellites in Earth orbit presents a growing challenge for space domain awareness. Detecting these objects, especially dim ones, is crucial for collision avoidance, tracking, and understanding the space environment. Traditional electro-optical sensors struggle with dim objects, particularly in the presence of bright backgrounds like sunlight or atmospheric turbulence. While radar and laser systems exist, optical solutions offer a more versatile and cost-effective approach. However, existing optical methods often face difficulties differentiating faint satellites from background noise (stars, etc.) due to similar apparent brightness and shape. This paper aims to enhance the detection of these dim objects by utilizing the unique capabilities of event-based cameras, which respond to changes in luminance rather than capturing full frames.

Methods

The core of this work is a hybrid deep learning architecture that fuses features from two sources:

  • Integrated Event-Based Frames: Event data, which is inherently sparse and asynchronous, is integrated over short time intervals to create 2D "frames." These are then processed by a conventional image feature extractor (specifically, DarkNet53, a backbone used in YOLOv3).
  • Point-Cloud Features: The raw event stream, represented as a sequence of (x, y, t, polarity) events, is treated as a 3D point cloud. This point-cloud data is fed into specialized feature extractors like PointNet, PointNet++, and PointConv, which are designed to handle unordered point sets and capture spatial relationships.

These two feature streams are then fused and passed through a YOLOv3 detection head to predict bounding boxes and object classes. A key innovation is the development of an event-based vision simulator that generates synthetic training data. This simulator allows for the optimization of sensor parameters (like contrast threshold) and scene conditions to maximize the event-based signal-to-noise ratio (EB-SNR), crucial for detecting dim targets.

Datasets

  • Simulated Dataset: A large synthetic dataset was generated using a combination of the Tensorflow-based SatSim space scene simulator and a custom event-based vision simulator. This simulator allows for control over various parameters like target magnitude, background brightness, field-of-view, and exposure time.
  • Real Dataset: A small, annotated dataset of real-world satellite observations from a Prophesee Gen 3 VGA event-based sensor was used for validation and to assess the simulator-to-real gap. This dataset contained 857 samples.

Key Insights and Ideas

  • Event Data's Temporal Advantage: The asynchronous, high-temporal-resolution nature of event data is particularly well-suited for capturing the subtle motion and changes of dim objects against a dynamic background.
  • Point-Cloud Features for Sparse Data: Treating event streams as point clouds allows specialized geometric deep learning models (like PointNet) to extract meaningful features from sparse, spatio-temporal data, which is challenging for traditional image-based methods.
  • Hybrid Fusion is Key: Simply using integrated event frames or point-cloud features alone is less effective than fusing both. The combination leverages the strengths of both image-like representations and the raw temporal information from events.
  • Simulation is Crucial for Training: The lack of large, diverse event-based datasets for satellite detection necessitates robust simulation. The authors developed a simulator that allows for parameter optimization and generation of training data that mimics real-world conditions.
  • Contrast Threshold is Paramount: The "contrast threshold" for event generation is a critical parameter that significantly impacts the detectability of dim objects. Optimizing this threshold, even in simulation, is vital for maximizing EB-SNR.

Key Results

  • The inclusion of point-cloud feature extractors (PointNet, PointNet++, PointConv) significantly improved detection performance compared to a frame-based only baseline.
  • Specifically, the hybrid models showed an increase in maximum F1 score and area under the precision-recall (auPR) curves. For instance, on the simulated dataset, the DarkNet-PointConv model achieved an auPR of 0.88152, compared to 0.87822 for the frame-based baseline. On real data, the improvement was even more pronounced, with DarkNet-PointNet++ achieving an auPR of 0.63179 compared to 0.5297 for the baseline.
  • The study highlighted that field-of-view and EB-SNR were strong indicators of model recall in simulation. Target velocity also showed a slight parabolic trend, suggesting that very slow or very fast targets are harder to detect.
  • The simulator-to-real gap was observed, with models trained on simulated data showing a performance drop when evaluated on real data. However, the simulation still proved to be a useful tool for experimentation and model generalization.

Limitations and Future Work

  • Sim-to-Real Gap: While the simulation is useful, discrepancies exist between simulated and real data, particularly in the ratio of positive to negative events and overall EB-SNR. The exact contrast threshold used in real-world collection was often unavailable, hindering a perfect match.
  • Limited Real Data: The real dataset used for validation was small (857 samples), limiting the scope of real-world evaluation.
  • Future Work:
    • Neural Network-based Contrast Threshold Tuning: Developing a more adaptive, neural network-based approach to tune the event-based contrast threshold in real-time could significantly improve detection and visibility.
  • Advanced Point-Cloud Algorithms: Exploring more sophisticated clustering and sampling algorithms for event streams could further enhance noise filtering and feature extraction.
  • Extensive Real Data Collection: Acquiring more extensive real-world data with varied conditions, parameters, and known contrast thresholds is essential for future simulator improvements and validation.
  • Architecture Exploration: Further exploration of hybrid architectures could potentially yield even better detection performance.

Summary updated

BibTeX
@inproceedings{salvatore2022learned,
  author = {Nikolaus Salvatore and Justin Fletcher},
  title = {{Learned Event-Based Visual Perception for Improved Space Object Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure