EMF: Event Meta Formers for Event-based Real-time Traffic Object Detection

Muhammad Ahmed Ullah Khan, Abdul Hannan Khan, Andreas DengelarXiv 2025 · DOI · arXiv

Abstract

Event cameras offer high speed and dynamic range for autonomous driving, but current transformer-based detectors are often too slow for real-time deployment. This paper introduces Event Meta Formers (EMF), a convolution-based architecture tailored for event data. By utilizing a MetaFormer structure and a novel Event Progression Extractor (EPE), EMF achieves state-of-the-art accuracy while significantly reducing inference time, reaching 9.1 ms per frame on standard hardware. This makes it the fastest deep learning model for event-based traffic object detection to date.

Introduction

Autonomous driving perception systems typically rely on standard RGB cameras, which struggle with motion blur at high speeds, low-light conditions, and high processing latency. Event cameras address these physical limitations by asynchronously recording "events" only when a pixel's intensity changes, providing microsecond temporal resolution and high dynamic range.

However, event data is fundamentally different from standard images, and traditional CNNs are not naturally suited for it. While recent state-of-the-art solutions use Recurrent Vision Transformers (RVTs), these models are computationally expensive and fail to exploit the unique temporal sparsity of event streams. EMF bridges this gap by providing a model that is both highly accurate and fast enough for real-time, resource-constrained autonomous driving.

The EMF Architecture

The core of the proposed system is the Event Meta Former (EMF) backbone. Unlike Vision Transformers that rely on computationally heavy self-attention, EMF adopts a MetaFormer structure using efficient convolutional alternatives.

Overview of the EMF architecture components, including the RepMixer, EMF Block, and the full backbone.

Key architectural innovations include:

  • Event Progression Extractor (EPE): To prevent spatial gradients from overwhelming temporal cues, the EPE uses point-wise convolutions at the start of the network to enrich and extract temporal progressions before spatial features dominate.
  • MetaFormer Structure: EMF replaces multi-axis self-attention with RepMixer (a token mixer) and ConvFFN (a channel mixer). This approach improves local feature extraction for event contours without the massive overhead of attention mechanisms.
  • No Patching: By avoiding the patching common in ViTs, EMF preserves natural spatial continuity and direct information paths.
  • Structural Reparameterization: The model utilizes train-time over-parameterization, merging multiple convolutions into one during inference to maximize throughput.

Experimental Evaluation

EMF was evaluated on two major automotive benchmarks: the Prophesee Gen1 dataset (39 hours of recordings at 304x240 resolution) and the Prophesee 1 Megapixel (1Mpx) dataset (14.6 hours at 1280x720 resolution). These datasets provide rigorous testing grounds for detecting cars, pedestrians, and two-wheelers in real-world driving scenarios.

Performance and Efficiency

EMF sets a new benchmark for both accuracy and speed in event-based detection. On the Gen1 dataset, it achieved 49.1 mAP, outperforming the previous state-of-the-art (RVT-B) by 1.9 mAP.

Comparison of EMF against state-of-the-art methods on Gen1 and 1Mpx datasets, highlighting mAP, inference time, and parameter count.

Beyond accuracy, the model's efficiency is its standout feature. EMF reduced inference time by 14-20% compared to RVT models, clocking in at just 9.1 ms per frame on an RTX 3090. It achieved these results with approximately 19% fewer parameters than RVT-B (14.9M vs 18.5M). Furthermore, cross-dataset evaluations showed that EMF generalizes better to unseen driving conditions, outperforming RVT by an average of 1.0 mAP when tested on datasets different from its training source.

Conclusion

The success of EMF demonstrates that processing temporal progressions before spatial features is vital for event-based vision. By replacing complex self-attention with convolution-based token mixers, EMF achieves superior local feature extraction with significantly lower computational costs. Future work may involve adapting the EPE concept to work directly on sparse point-cloud representations to further leverage the inherent sparsity of event data.

Summary updated

BibTeX
@misc{khan2025emf,
  author = {Muhammad Ahmed Ullah Khan and Abdul Hannan Khan and Andreas Dengel},
  title = {{EMF: Event Meta Formers for Event-based Real-time Traffic Object Detection}},
  year = {2025},
  eprint = {2504.04124},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.04124},
}
Figure