CREST: An Efficient Conjointly-trained Spike-driven Framework for Event-based Object Detection Exploiting Spatiotemporal Dynamics

Ruixin Mao, Aoyu Shen, Lin Tang, Jun ZhouAAAI 2025 · DOI

TL;DR

This paper introduces CREST, a novel spike-driven framework for event-based object detection that leverages spatiotemporal dynamics for improved accuracy and energy efficiency. By proposing a conjoint learning rule and a multi-scale event integrator, CREST addresses limitations in existing Spiking Neural Networks (SNNs) for this task, achieving state-of-the-art performance with significantly reduced energy consumption.

What Problem Does This Paper Address?

Event-based cameras offer advantages like high temporal resolution, wide dynamic range, and low power consumption, making them ideal for challenging scenarios like high-speed and low-light object detection. Spiking Neural Networks (SNNs) are a natural fit for these cameras due to their event-driven nature and potential for energy efficiency. However, current SNNs face several challenges:

  • Inefficient Training: Deep SNNs suffer from gradient vanishing and high computational complexity, hindering their practical application.
  • Handling Spatiotemporal Features: Existing methods struggle to effectively integrate multi-scale spatiotemporal information from event data, leading to redundancy and reduced accuracy.
  • Energy Consumption: While SNNs are theoretically energy-efficient, current training methods and architectures can still be power-hungry, especially on hardware not optimized for SNNs.
  • Loss Functions: Standard loss functions, often adapted from frame-based approaches, don't fully exploit the unique spatiotemporal nature of event data.

This paper aims to bridge these gaps by proposing a more efficient and effective SNN framework for event-based object detection.

Methods

CREST introduces several key innovations:

  • Conjoint Learning Rule (CLR): This novel rule accelerates SNN learning and mitigates gradient vanishing. It employs a surrogate neural network (DL-Net) with discrete-level activations to mimic spike train patterns, replacing complex spatiotemporal gradient calculations. CLR supports dual operation modes for flexible implementation on GPUs/TPUs or dedicated SNN hardware.
  • Multi-scale Spatiotemporal Event Integrator (MESTOR): This module integrates event features from multiple scales. It extracts and clusters continuous spatiotemporal events while reducing redundant background and noise, leading to a more robust representation.
  • Spatiotemporal IoU (ST-IoU) Loss: This new loss function combines a novel "Spiking-IoU" (which quantifies spike density differences between predicted and ground truth boxes) with the Complete IoU (CIoU) loss. This approach better leverages the spatiotemporal continuity of event data for more precise detection.
  • FSN-SNN Model: The framework utilizes Few-Spikes Neurons (FSNs) as its core computational unit. FSNs are known for their high encoding efficiency and compatibility with specialized SNN hardware like STELLAR.

Datasets

The paper evaluates CREST on the following datasets:

  • NCARs: Used for object recognition.
  • Gen1 and PKU-Vidar-DVS: Used for object detection, particularly noted for their challenging extreme lighting conditions for event-based data.

Key Insights and Ideas

  • Conjoint Learning for SNNs: The idea of a "conjoint" learning rule, where a surrogate network mimics spike patterns, is a clever way to leverage gradient-based optimization without directly backpropagating through non-differentiable spikes, thus improving training efficiency and stability in deep SNNs.
  • MESTOR's Multi-Scale Integration: The approach of explicitly integrating features across different spatial and temporal scales (DST, DS, DT) is crucial for capturing the rich spatiotemporal dynamics present in event data, which is often lost in simpler event aggregation methods.
  • Spatiotemporal IoU for Event Data: Adapting IoU loss to consider spike density (Spiking-IoU) and combining it with CIoU (ST-IoU) is a significant step towards creating loss functions tailored for the unique characteristics of event-based vision.
  • Hardware-Aware Design: The dual operation modes for CLR highlight a practical consideration for deploying SNNs, acknowledging the differences between general-purpose hardware (GPUs/TPUs) and specialized SNN accelerators.

Key Results

CREST demonstrates impressive performance across various metrics:

  • Accuracy: Achieves state-of-the-art accuracy on object recognition and detection tasks, outperforming existing SNN and ANN methods on datasets like NCARs, Gen1, and PKU-Vidar-DVS. For instance, on Gen1, it improves mAP50 from 0.59 to 0.63.
  • Energy Efficiency: Shows significant energy savings, with up to 100x improvement compared to state-of-the-art SNN algorithms. Even the smallest CREST network achieves very low energy consumption (e.g., 0.01mJ on NCARs).
  • Efficiency: The conjoint learning rule reduces training time per epoch by up to 3.4x. The use of FSNs also contributes to lower firing rates, further enhancing efficiency.

Limitations and Future Work

  • Limitations:
    • The paper notes that while FSNs have negligible decay energy costs, LIF-based SNNs still incur energy costs for membrane potential decay, which is a general challenge for LIF models.
  • The paper implicitly acknowledges that the performance of SNNs can still be sensitive to hyperparameters like the time window (K) and the weighting of IoU components (a, b), as shown in ablation studies.
  • Future Work:
    • The authors suggest that CREST provides an efficient solution for event-based object detection suitable for SNN hardware implementation, implying future work could focus on further optimizing for specific hardware platforms.
  • The exploration of different IoU weights (a and b) in ST-IoU suggests potential for further fine-tuning this loss function for optimal performance.

Summary updated

BibTeX
@inproceedings{mao2025crest,
  author = {Ruixin Mao and Aoyu Shen and Lin Tang and Jun Zhou},
  title = {{CREST: An Efficient Conjointly-trained Spike-driven Framework for Event-based Object Detection Exploiting Spatiotemporal Dynamics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i6.32649},
}
Figure