Abstract
Event cameras capture asynchronous intensity changes with microsecond latency, offering significant advantages for high-speed robotics and autonomous driving. However, existing deep learning models often bottleneck this performance by processing events in fixed-rate frames. We present the Hierarchical Neural Memory Network (HMNet), an architecture designed to natively exploit the multi-scale temporal nature of event data. By utilizing a hierarchy of latent memories operating at different frequencies, HMNet processes fast-moving objects with high temporal resolution while updating global context more efficiently. Our approach reduces latency by 40–50% compared to standard recurrent baselines and achieves state-of-the-art results on benchmarks for semantic segmentation, object detection, and depth estimation.
Introduction
Standard cameras capture full frames at fixed intervals, which can lead to motion blur or redundant data in static scenes. In contrast, event cameras record only pixel-level intensity changes asynchronously. While this provides high temporal resolution and low latency, most current AI models fail to fully leverage these properties. They typically group events into fixed-duration "frames" and process them using standard Recurrent Neural Networks (RNNs) or CNNs. This "fixed-rate" approach treats all parts of a scene with the same temporal resolution, either wasting computation on static backgrounds or losing the low-latency benefits for dynamic objects.
The Hierarchical Neural Memory Network (HMNet)
The core innovation of HMNet is its multi-rate latent memory structure. Instead of a single processing rate, the network maintains a stack of memories ($z_1, z_2, \dots, z_L$) that operate at varying frequencies:
- Low-level memory ($z_1$): Operates at the highest frequency to capture rapid, local changes essential for tracking fast-moving objects.
- High-level memories ($z_2, z_3$): Operate at lower frequencies, extracting global semantic context that changes more slowly over time.
Information is exchanged between these levels through "Up-write" and "Down-write" operations. This allows the network to maintain a high-speed response for dynamic elements while grounding them in a stable understanding of the broader scene. To integrate sparse event data into these dense memory cells, we developed Event Sparse Cross Attention (ESCA). ESCA uses a window-based attention mechanism and includes a trainable "event gate" that filters out noise, ensuring that only meaningful intensity changes update the stored representation.
Furthermore, HMNet supports asynchronous sensor fusion. Standard RGB images can be injected into the high-level memory at their native (slower) frame rates, effectively "anchoring" the high-frequency event features with rich visual detail without forcing the entire network to slow down.
Experimental Results
We evaluated HMNet across several dense prediction tasks using established benchmarks:
- Semantic Segmentation: On the DSEC-Semantic benchmark, HMNet-L3 achieved a state-of-the-art mIoU of 57.1.
- Object Detection: In the GEN1 dataset, HMNet-B1 demonstrated competitive accuracy with a remarkably low latency of 2.4ms.
- Monocular Depth Estimation: Evaluated on the MVSEC and Eventscape datasets, the model showed robust performance.
Key findings include a 40–50% reduction in latency compared to conventional recurrent baselines. The hierarchical structure itself proved vital; a multi-level memory configuration yielded a +3.8% mIoU improvement over a single-level version. Additionally, the architecture's inherent parallelism allows for multi-GPU scaling, further reducing processing time for complex models.
Discussion and Conclusion
HMNet demonstrates that temporal redundancy is a significant inefficiency in current event-processing models. By updating high-level features only when necessary, computation is saved without sacrificing accuracy. While the model shows some sensitivity to inference time steps—with performance degrading if steps exceed 10ms—it represents a major step toward real-time, high-speed vision. Future work will explore extending this hierarchical memory approach to other domains, such as video action recognition and the integration of additional sensors like LiDAR.
Summary updated