Within-Camera Multilayer Perceptron DVS Denoising

Antonio Rios-Navarro, S. Guo, G. Abarajithan, K. Vijayakumar, Alejandro Linares-Barranco, Thea Aarrestad, Ryan Kastner, Tobi DelbruckWorkshops @ CVPR 2023 · DOI

TL;DR

This paper introduces a novel hardware-accelerated Multilayer Perceptron (MLP) denoiser for event cameras, enabling efficient, in-camera noise filtering. By implementing a quantized MLP directly on-chip (FPGA or ASIC), the system significantly reduces data rates and host processing power, opening up new possibilities for low-power, always-on event-based vision applications.

What Problem Does This Paper Address?

Event cameras, like the Dynamic Vision Sensor (DVS), offer significant advantages in terms of high dynamic range and low latency compared to traditional frame-based cameras. However, they suffer from a fundamental issue: Background Activity (BA) noise, which can drastically increase the data rate, especially in low-light conditions. This noise not only consumes bandwidth but also burdens the downstream processing units (like a host computer) with unnecessary data. While software-based denoising exists, it often negates the power and latency benefits of event cameras by requiring significant processing power on a separate host. The challenge is to perform effective denoising directly within the camera hardware to reduce data rates and power consumption without sacrificing accuracy.

Methods

The core of this work is the development and hardware implementation of a Multilayer Perceptron (MLP) denoiser, termed the MLPF. This MLP is designed to process events in real-time, operating on a small neighborhood of pixels around each incoming event.

Key methodological aspects include:

  • Spatiotemporal Feature Extraction: The MLPF utilizes a neighborhood of pixels and their associated timestamps to infer the "age" of events. This spatiotemporal context is crucial for distinguishing genuine signal events from random noise.
  • Quantized Neural Network: To enable efficient hardware implementation, the MLP's weights and activations are heavily quantized to 4 bits. This significantly reduces the computational and memory requirements.
  • Hardware Implementations: The paper presents two primary hardware implementations:
  • FPGA Implementation: Using the hls4ml library, the MLP is translated into High-Level Synthesis (HLS) code, which is then compiled for an FPGA. This approach allows for rapid prototyping and integration.
  • ASIC Implementation: A custom RTL (SystemVerilog) implementation of the MLP is developed for a 65nm digital process. This offers the highest potential for power efficiency and performance.
  • Quantization-Aware Training: To mitigate the accuracy loss due to aggressive quantization, the MLP is trained using QKERAS, a library that allows quantization to be incorporated directly into the training process.

Datasets

The paper utilizes datasets previously established in [5]:

  • Hotel-bar dataset: This dataset is described as representing sparse surveillance scenarios.
  • Driving dataset: This dataset is characterized as representing dense mobile robotics event camera output.

Both datasets were augmented with simulated and pre-recorded Background Activity (BA) noise events to create challenging denoising scenarios. The authors note that these datasets were chosen to cover a range of applications.

Key Insights and Ideas

  • In-Camera Denoising is Crucial for Low-Power Event Vision: The most significant insight is that performing denoising directly within the event camera is essential for unlocking the full potential of event-based vision in power-constrained applications. By reducing the data rate at the source, the energy consumed by communication and host processing can be dramatically lowered.
  • Quantized MLPs Offer a Sweet Spot for Hardware: The paper demonstrates that a heavily quantized MLP (4-bit weights and activations) can achieve denoising accuracy nearly indistinguishable from its floating-point counterpart. This is a critical finding, as it bridges the gap between the accuracy of deep learning models and the efficiency demands of hardware.
  • Spatiotemporal Context is Key for Noise Discrimination: The MLPF's ability to leverage the temporal information (event age) within a local neighborhood is a core idea that allows it to effectively discriminate between signal and noise, outperforming simpler correlation-based methods.
  • Hardware Implementations Enable Real-Time Performance: The successful implementation on both FPGA and ASIC proves that this MLPF denoiser can operate at event rates relevant to modern event cameras (up to 25 MHz for the ASIC).

Key Results

  • Accuracy: The 4-bit quantized MLPF achieves an Area Under the Curve (AUC) of 0.87 on the driving dataset, very close to the floating-point model's AUC of 0.88. This indicates minimal accuracy degradation due to quantization.
  • Data Rate Reduction: The MLPF can reduce the data rate from 10MB/s to 100kB/s in low-light scenes, and the host processor interrupt rate from 1 kHz to less than 10 Hz.
  • FPGA Performance: The FPGA implementation processes each event in 10 clock cycles (100 ns) at 100 MHz, enabling denoising at event rates up to 10 MHz. It uses 3.5k flip-flops and 11.5k LUTs.
  • ASIC Performance: The ASIC implementation in 65nm technology for a 346x260 pixel camera occupies 4.3mm² and consumes 4nJ of energy per event at event rates up to 25 MHz. The estimated standby power is around 35 mW.
  • Comparison to Other Denoisers: The MLPF demonstrates superior AUC accuracy compared to other low-cost denoisers like STCF and BAF, particularly at higher noise rates.

Limitations and Future Work

  • SRAM Dominance in ASIC Area: The paper notes that for the ASIC implementation, the SRAM used for storing the Timestamp+Polarity Image (TPI) dominates the chip area. While the logic area is small, the memory footprint is significant.
  • Latency Trade-offs: The ASIC implementation has a longer latency in clock cycles (33 cycles) compared to the FPGA (10 cycles), although its higher clock frequency results in a faster overall event classification time (40 ns vs. 100 ns).
  • Future Work: The authors suggest that integrating lower-leakage SRAM in the ASIC design could further reduce standby power by an order of magnitude. They also highlight the potential for these hardware-accelerated denoisers to enable new "always-on" event camera applications under scavenged or battery power. The integration of the MLPF directly onto event camera chips or in the same package is also a promising direction.

Summary updated

BibTeX
@inproceedings{riosnavarro2023within,
  author = {Antonio Rios-Navarro and S. Guo and G. Abarajithan and K. Vijayakumar and Alejandro Linares-Barranco and Thea Aarrestad and Ryan Kastner and Tobi Delbruck},
  title = {{Within-Camera Multilayer Perceptron DVS Denoising}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00409},
}
Figure