Event cameras, or Dynamic Vision Sensors (DVS), capture the visual world fundamentally differently from traditional frame-based cameras. Instead of capturing global images at fixed intervals, each pixel operates independently and asynchronously, emitting a "spike" or event only when it detects a change in log-intensity brightness. As detailed in the comprehensive survey by Guillermo Gallego et al., this design yields microsecond-level latency, massive dynamic range, and ultra-low power consumption.
However, a severe bottleneck arises when these sparse streams meet traditional computer vision pipelines. Because standard deep learning models and Von Neumann hardware are optimized for dense, synchronous grids, a common workaround is to accumulate events over a time window into artificial, image-like "frames." This event-to-frame conversion introduces a heavy computational tax, reintroducing data redundancy and latency, which effectively nullifies the sensor's core benefits. To unlock the true potential of neuromorphic sensing, a growing body of research focuses on native asynchronous processing.
Why Frame Aggregation Destroys Efficiency
To understand why converting events to frames is highly inefficient, consider what happens inside a standard Convolutional Neural Network (CNN) processing an aggregated event frame. Even if a scene is highly sparse—meaning only 1% of the pixels contain motion events—a standard convolutional layer will perform dense matrix multiplications across the entire spatial grid.
[Sparse Events] ---> (Frame Aggregation) ---> [Dense Matrix] ---> (Standard Convolution)
(Efficient) | (Inefficient: computes on zeros)
v
Introduces Latency &
Wastes Power
Worse, standard convolutions act as low-pass filters that "dilate" active pixels. After passing through just a few layers, the initial structural sparsity of the event stream is completely lost, forcing the network to waste power computing on empty spatial regions. Furthermore, waiting to accumulate enough events to form a coherent frame introduces artificial millisecond-scale delays. As Alejandro Linares-Barranco et al. point out, these operating system and scheduling delays can cripple ultra-high-speed robotic control loops that require sub-millisecond responses.
Algorithmic Solutions: Sparse-Aware Architectures
To preserve the sensor's natural efficiency, researchers have designed neural network layers that respect and maintain sparsity throughout the computational graph.
Submanifold Sparse Convolutions
Instead of computing everywhere, submanifold sparse convolutions restrict the convolution's output strictly to locations that were already active in the input layer. Daniel Gehrig et al. demonstrated that by leveraging these sparse operations, standard high-accuracy CNN architectures can process event data asynchronously. This maintains the exact same accuracy as dense models while cutting computational complexity (FLOPs) by up to 20x.
Patch-Based Attention
For attention-based models, standard vision transformers suffer from quadratic complexity relative to the number of tokens. To circumvent this, Alberto Sabater et al. developed the Event Transformer (EvT). EvT processes data by focusing exclusively on "activated patches" where motion occurs, combining a sparse-aware representation with a compact architecture to eliminate redundant calculations on static background regions.
Differentiable Recurrent Surfaces
To capture temporal dynamics without frame-based binning, networks must maintain a continuous internal memory. Marco Cannici et al. introduced Matrix-LSTM, which replaces hand-crafted event-to-frame heuristics with a grid of LSTM cells that learn task-specific representations directly from asynchronous data. Similarly, Etienne Perot et al. demonstrated with their Recurrent Event-camera Detector (RED) that memory-aware recurrent models can operate directly on raw event streams, achieving high-resolution object detection while remaining highly power-efficient.
Hardware-Software Co-Design: Fully Neuromorphic Pipelines
True asynchronous efficiency cannot be achieved by software alone; the underlying hardware must also match the event-driven nature of the data. Traditional CPUs and GPUs operate on global clocks and physically separate memory from computation, creating a Von Neumann bottleneck when handling sparse spikes.
Traditional (Von Neumann): [Global Clock] ---> [Fetch Memory] ---> [Compute] ---> [Store]
Neuromorphic (Spike-Driven): [Event Occurs] ---> [Asynchronous Local Update]
To bypass this, researchers are moving toward end-to-end neuromorphic pipelines where sensing, processing, and control occur entirely within the event domain. Alexander Andreopoulos et al. demonstrated this paradigm shift by building a fully event-based stereo vision system entirely on neuromorphic hardware, proving that moving away from frame-based architectures yields massive throughput and power advantages.
Spiking Neural Networks (SNNs) running on neuromorphic processors, such as Intel's Loihi chip, provide a natural substrate for this paradigm. Because SNNs only compute when a spike occurs, they feature zero idle power consumption. For example:
- Alpha Renner et al. implemented Dynamic Neural Fields (DNFs) as an SNN on Loihi to track objects asynchronously, ensuring memory and processing remain co-located.
- Junjie Jiang et al. built a fully asynchronous spike-driven pipeline for mobile robot dodging on Loihi, consuming less than 2% of the energy required by traditional frame-based methods.
When specialized neuromorphic chips are unavailable, alternative hardware strategies can still eliminate the hybrid translation bottleneck. Alejandro Linares-Barranco et al. developed an open-source FPGA library that processes events in under 300 nanoseconds by moving filtering and feature extraction directly into hardware gates. On the cutting edge of analog computing, Yue Yang et al. utilized scalable memristive neurons to mimic biological escape reflexes, achieving obstacle avoidance with 50x lower latency by eliminating the digital translation step entirely.
By aligning sparse-aware software algorithms with asynchronous, event-driven hardware, this research direction successfully eliminates the event-to-frame conversion bottleneck, delivering on the original promise of neuromorphic engineering: perception at the speed of light, on a milliwatt budget.
Key Foundational Work
A Low Power, High Throughput, Fully Event-Based Stereo System (2018) — contributed 1 concept elements to this direction.
Managing Burstiness and Scalability in Event-Driven Models on the SpiNNaker Neuromimetic System (2011) — contributed 1 concept elements to this direction.