Recognizing Ultra-High-Speed Moving Objects with Bio-Inspired Spike Camera

Junwei Zhao, Shiliang Zhang, Zhaofei Yu, Tiejun Huang

Introduction

Capturing and recognizing objects moving at extreme velocities presents a significant challenge for traditional computer vision. Standard frame-based cameras, typically operating at 30 or 60 frames per second (fps), suffer from severe motion blur when recording fast-moving objects like racing cars or projectiles. While bio-inspired spike cameras—which utilize independent pixels that fire digital spikes upon reaching a light accumulation threshold—offer sampling rates up to 1,000 times faster than human vision, they are not immune to physical limitations.

Overview of vision sensors and data representation for fast-moving objects.

This research identifies and theoretically analyzes "spiking motion blur," a phenomenon occurring when object speeds exceed the hardware's sampling capacity (typically over 500 km/h). To address this, the authors propose a Temporal-Spatial Context (TSC) learning framework and introduce the UHSR dataset, the first real-world benchmark for ultra-high-speed neuromorphic recognition.

Theoretical Analysis of Spiking Motion Blur

By applying the Shannon sampling theorem to the physical constraints of CMOS technology, the researchers established that spiking motion blur is driven by two primary factors: temporal undersampling and spatial misalignment. Temporal undersampling occurs when the spike rate is insufficient to capture rapid changes, while spatial misalignment happens when an object moves across multiple pixels before a single spike can be triggered.

Comparison of motion blur in frame cameras vs. spike cameras at high and ultra-high speeds.

A key theoretical insight is that the "speed limit" ($V^*$) of a spike camera is directly proportional to scene brightness. In high-luminance environments, pixels accumulate charge faster, effectively increasing the camera's temporal resolution and allowing it to track faster objects without blur.

The Temporal-Spatial Context (TSC) Learning Framework

To recover information from blurred spike streams, the authors developed the TSC framework, which integrates three specialized modules:

  1. Multi-span Temporal-Dilated Convolution (MTDC): This module employs dilated convolutions across the temporal axis to capture motion continuity across varying time scales.
  2. Residual Deformable Convolution (RDC): To combat spatial smearing, RDC uses deformable kernels that adapt their shape to match the distorted patterns of fast-moving objects.
  3. Cross-Attention Feature Aggregation (CAFA): An attention mechanism that intelligently fuses temporal and spatial features, prioritizing the most informative segments of the spike stream.

Experimental Results and Datasets

The study introduces the Ultra-High-Speed Recognition (UHSR) dataset, containing 12,000 real-world samples of objects moving at equivalent speeds of 500–700 km/h.

In comparative evaluations, the TSC method demonstrated state-of-the-art performance:

  • Accuracy: On the 10-class U-CIF dataset, TSC achieved 73.2% accuracy, significantly outperforming previous spike-based methods like DMER (67.8%) and TSR (65.9%).
  • Scalability: On the 101-class U-CAL dataset, it reached 67.5%, marking an 8.3% improvement over baseline models.
  • Efficiency: The model maintained high accuracy even with a minimal 1ms input window, proving its capability for near-instantaneous decision-making.

Conclusion

This work provides a foundational theoretical understanding of the limits of neuromorphic spike cameras. By identifying the relationship between brightness, hardware constraints, and motion blur, and providing a robust learning framework in TSC, the researchers have pushed the boundaries of high-speed vision. Future work will focus on transitioning these methods from controlled laboratory settings to complex real-world environments and addressing the underlying hardware bottlenecks in CMOS photo-electric conversion.

Summary updated

Abstract

Bio-inspired spike camera mimics the sampling principle of primate fovea. It presents high temporal resolution and dynamic range, showing great promise in fast-moving object recognition. However, the physical limit of CMOS technology in spike cameras still hinders their capability of recognizing ultra-high-speed moving objects, e.g., extremely fast motions cause blur during the imaging process of spike cameras. This paper presents the first theoretical analysis for the causes of spiking motion blur and proposes a robust representation that addresses this issue through temporal-spatial context learning. The proposed method leverages multi-span feature aggregation to capture temporal cues and employs residual deformable convolution to model spatial correlation among neighbouring pixels. Additionally, this paper contributes an original real-captured spiking recognition dataset consisting of 12,000 ultra-high-speed (equivalent speed > 500 km/h) moving objects. Experimental results show that the proposed method achieves 73.2% accuracy in recognizing 10 classes of ultra-high-speed moving objects, outperforming all existing spike-based recognition methods. Resources will be available at https://github.com/Evin-X/UHSR.
BibTeX
@inproceedings{zhao2024recognizing,
  author = {Junwei Zhao and Shiliang Zhang and Zhaofei Yu and Tiejun Huang},
  title = {{Recognizing Ultra-High-Speed Moving Objects with Bio-Inspired Spike Camera}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28579},
}
Figure