Hierarchical Neural Memory Network for Low Latency Event Processing

Ryuhei Hamaguchi, Yasutaka Furukawa, Masaki Onishi, Ken Sakurada

Abstract

Event cameras capture asynchronous intensity changes with microsecond latency, offering significant advantages for high-speed robotics and autonomous driving. However, existing deep learning models often bottleneck this performance by processing events in fixed-rate frames. We present the Hierarchical Neural Memory Network (HMNet), an architecture designed to natively exploit the multi-scale temporal nature of event data. By utilizing a hierarchy of latent memories operating at different frequencies, HMNet processes fast-moving objects with high temporal resolution while updating global context more efficiently. Our approach reduces latency by 40–50% compared to standard recurrent baselines and achieves state-of-the-art results on benchmarks for semantic segmentation, object detection, and depth estimation.

Introduction

Standard cameras capture full frames at fixed intervals, which can lead to motion blur or redundant data in static scenes. In contrast, event cameras record only pixel-level intensity changes asynchronously. While this provides high temporal resolution and low latency, most current AI models fail to fully leverage these properties. They typically group events into fixed-duration "frames" and process them using standard Recurrent Neural Networks (RNNs) or CNNs. This "fixed-rate" approach treats all parts of a scene with the same temporal resolution, either wasting computation on static backgrounds or losing the low-latency benefits for dynamic objects.

Conceptual comparison between conventional fixed-rate RNNs and the proposed variable-rate HMNet.

The Hierarchical Neural Memory Network (HMNet)

The core innovation of HMNet is its multi-rate latent memory structure. Instead of a single processing rate, the network maintains a stack of memories ($z_1, z_2, \dots, z_L$) that operate at varying frequencies:

  • Low-level memory ($z_1$): Operates at the highest frequency to capture rapid, local changes essential for tracking fast-moving objects.
  • High-level memories ($z_2, z_3$): Operate at lower frequencies, extracting global semantic context that changes more slowly over time.
The HMNet architecture featuring hierarchical memories ($z_1, z_2, z_3$) and the task head.

Information is exchanged between these levels through "Up-write" and "Down-write" operations. This allows the network to maintain a high-speed response for dynamic elements while grounding them in a stable understanding of the broader scene. To integrate sparse event data into these dense memory cells, we developed Event Sparse Cross Attention (ESCA). ESCA uses a window-based attention mechanism and includes a trainable "event gate" that filters out noise, ensuring that only meaningful intensity changes update the stored representation.

Furthermore, HMNet supports asynchronous sensor fusion. Standard RGB images can be injected into the high-level memory at their native (slower) frame rates, effectively "anchoring" the high-frequency event features with rich visual detail without forcing the entire network to slow down.

Experimental Results

We evaluated HMNet across several dense prediction tasks using established benchmarks:

  • Semantic Segmentation: On the DSEC-Semantic benchmark, HMNet-L3 achieved a state-of-the-art mIoU of 57.1.
  • Object Detection: In the GEN1 dataset, HMNet-B1 demonstrated competitive accuracy with a remarkably low latency of 2.4ms.
  • Monocular Depth Estimation: Evaluated on the MVSEC and Eventscape datasets, the model showed robust performance.

Key findings include a 40–50% reduction in latency compared to conventional recurrent baselines. The hierarchical structure itself proved vital; a multi-level memory configuration yielded a +3.8% mIoU improvement over a single-level version. Additionally, the architecture's inherent parallelism allows for multi-GPU scaling, further reducing processing time for complex models.

Discussion and Conclusion

HMNet demonstrates that temporal redundancy is a significant inefficiency in current event-processing models. By updating high-level features only when necessary, computation is saved without sacrificing accuracy. While the model shows some sensitivity to inference time steps—with performance degrading if steps exceed 10ms—it represents a major step toward real-time, high-speed vision. Future work will explore extending this hierarchical memory approach to other domains, such as video action recognition and the integration of additional sensors like LiDAR.

Summary updated

Abstract

This paper proposes a low latency neural network architecture for event-based dense prediction tasks. Conventional architectures encode entire scene contents at a fixed rate regardless of their temporal characteristics. Instead, the proposed network encodes contents at a proper temporal scale depending on its movement speed. We achieve this by constructing temporal hierarchy using stacked latent memories that operate at different rates. Given low latency event steams, the multi-level memories gradually extract dynamic to static scene contents by propagating information from the fast to the slow memory modules. The architecture not only reduces the redundancy of conventional architectures but also exploits long-term dependencies. Furthermore, an attention-based event representation efficiently encodes sparse event streams into the memory cells. We conduct extensive evaluations on three event-based dense prediction tasks, where the proposed approach outperforms the existing methods on accuracy and latency, while demonstrating effective event and image fusion capabilities. The code is available at https://hamarh.github.io/hmnet/
BibTeX
@inproceedings{hamaguchi2023hierarchical,
  author = {Ryuhei Hamaguchi and Yasutaka Furukawa and Masaki Onishi and Ken Sakurada},
  title = {{Hierarchical Neural Memory Network for Low Latency Event Processing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure