Near-Chip Dynamic Vision Filtering for Low-Bandwidth Pedestrian Detection

Anthony Bisulco, Fernando Cladera Ojeda, Volkan Isler, Daniel Dongyuel Lee

Introduction

Dynamic Vision Sensors (DVS), often called "event cameras," represent a paradigm shift in visual sensing. Unlike traditional cameras that capture full frames at fixed intervals, DVS pixels only report changes in light intensity. This event-driven nature makes them inherently low-power and high-speed. However, deploying these sensors in long-term IoT monitoring scenarios (using networks like LoRa or NB-IoT) remains challenging because raw event streams can still exceed available bandwidth. Furthermore, streaming raw visual data to central servers poses significant privacy risks.

Comparison between the proposed low-bandwidth DVS architecture and conventional high-bandwidth image sensor systems.

This paper proposes a two-stage pipeline that addresses these issues by performing aggressive near-chip filtering followed by efficient edge-based detection. This "middle-ground" architecture compresses data near the sensor while maintaining enough information for accurate pedestrian identification.

System Architecture and Methods

The proposed architecture splits processing between a near-chip FPGA and a low-power microcontroller.

1. The Filtering Module (FPGA)

The filtering module resides close to the sensor and processes raw DVS events through four distinct stages to minimize data volume:

  • Event Parsing: Converts raw sensor signals into spatial coordinates and polarity $(x, y, p)$.
  • Coincidence Detection: Acts as a spatial-temporal denoiser, discarding events that do not have neighbors in close proximity, effectively removing background noise.
  • Aggregation & Subsampling: Events are grouped over time, and an $8 \times 8$ max-pooling operation reduces spatial resolution. This step is intentionally lossy to preserve privacy; the resulting "blobs" are sufficient for AI detection but insufficient for human identification.
  • Huffman Coding: Applies lossless compression to the resulting data packets before transmission.

2. The Detection Module (Microcontroller)

The compressed data is transmitted to an STM32F4 microcontroller running a Binary Neural Network (BNN). BNNs are ideal for edge devices because they use weights of +1 or -1, allowing complex multiplications to be replaced by fast bitwise operations (AND and popcount). This hardware-software co-design enables real-time inference on extremely constrained hardware, achieving a detection every 450ms.

Key Results

The system was evaluated using a custom DVS dataset recorded in an office environment. The results demonstrate a significant breakthrough in balancing performance, power, and bandwidth.

The four-stage filtering pipeline achieves a 99.6% reduction in data bandwidth while preserving essential features for detection.
  • Bandwidth Efficiency: The pipeline reduces the raw event stream from 22.25 Mb/s to a filtered bandwidth of just 74.58 kb/s—a total reduction of 99.6%. This makes the stream compatible with low-power wide-area network (LPWAN) standards.
  • Detection Accuracy: Despite the aggressive compression, the BNN maintains an 83% F1 score for pedestrian detection.
  • Power Consumption: The entire near-chip architecture consumes only 91.1 mW, making it orders of magnitude more efficient than traditional GPU or high-end CPU-based edge AI solutions which can consume upwards of 3W.

Conclusion and Future Work

By combining near-chip FPGA filtering with edge-based BNNs, this research provides a viable path for privacy-preserving, low-power IoT surveillance. While the current 450ms inference latency is suitable for office monitoring, future iterations could target higher-speed applications by integrating the components into a single Application Specific Integrated Circuit (ASIC). This work highlights the power of hardware-software co-design in overcoming the bandwidth and privacy barriers of modern visual sensing.

Summary updated

Abstract

This paper presents a novel end-to-end system for pedestrian detection using Dynamic Vision Sensors (DVSs). We target applications where multiple sensors transmit data to a local processing unit, which executes a detection algorithm. Our system is composed of (i) a near-chip event filter that compresses and denoises the event stream from the DVS, and (ii) a Binary Neural Network (BNN) detection module that runs on a low-computation edge computing device (in our case a STM32F4 microcontroller). We present the system architecture and provide an end-to-end implementation for pedestrian detection in an office environment. Our implementation reduces transmission size by up to 99.6% compared to transmitting the raw event stream. Our detector is able to perform a detection every 450 ms, with an overall testing F1 score of 83%. The low bandwidth and energy properties of our system make it ideal for IoT applications.
BibTeX
@inproceedings{bisulco2020near,
  author = {Anthony Bisulco and Fernando Cladera Ojeda and Volkan Isler and Daniel Dongyuel Lee},
  title = {{Near-Chip Dynamic Vision Filtering for Low-Bandwidth Pedestrian Detection}},
  booktitle = {ISVLSI},
  year = {2020},
  url = {https://openreview.net/forum?id=yjNM48Z84l},
}
Figure