Bridging Asynchronous Event Streams and Dense Neural Networks

18 concepts18 papers1 proposal

Traditional computer vision is built on a simple assumption: the world can be captured as a sequence of static, dense frames sampled at regular time intervals. Modern deep learning architectures, from Convolutional Neural Networks (CNNs) to Transformers, are fundamentally hardwired for this format. They expect data organized into neat, synchronous, grid-like tensors of shape $C \times H \times W$.

Event-based cameras (or bio-inspired neuromorphic sensors) completely break this assumption. Instead of capturing full frames at a fixed frame rate, each pixel operates independently and asynchronously. When a pixel detects a change in log-intensity exceeding a certain threshold, it immediately emits an "event"—a sparse tuple recording the spatial coordinates, a microsecond-accurate timestamp, and the polarity (direction) of the brightness change: $(x, y, t, p)$.

This creates a fundamental architectural incompatibility. We have a sensor capable of microsecond latency and high dynamic range, but its output is a sparse, irregularly timed point cloud that cannot be directly fed into standard deep learning models. To unlock the potential of event cameras for downstream tasks like semantic segmentation, optical flow, and object recognition, researchers must design representations that bridge this asynchronous-to-synchronous gap without sacrificing the unique temporal advantages of the sensor.

The Naive Solution: Accumulation and Frame Collapsing

The most intuitive way to resolve this incompatibility is to group events over a fixed time window $\Delta t$ and collapse them into a standard 2D grid.

In early frameworks, this was done using handcrafted statistical heuristics. For instance, the authors of EV-FlowNet accumulated events into a compact 4-channel image representation (counting positive and negative events, alongside their most recent timestamps) to enable self-supervised optical flow estimation. Similarly, the authors of EV-SegNet built a 6-channel representation capturing the statistical distribution of events over time to feed a semantic segmentation network.

Raw Event Stream (x, y, t, p) ───> [ Accumulation Window Δt ] ───> Dense 2D Grid (C × H × W)

This strategy immediately solves the compatibility problem. As demonstrated by Roshan Gopalakrishnan et al. in Classifying neuromorphic data using a deep learning framework for image classification, simply collapsing spike-timing data into static images allows practitioners to leverage powerful pre-trained frame-based networks like VGG-16, easily outperforming specialized neuromorphic algorithms at the time.

However, this handcrafted collapsing comes with a severe theoretical cost: it destroys the microsecond-level temporal resolution of the sensor. If two events occur at the same pixel within the same window $\Delta t$—one at the beginning and one at the very end—a simple accumulation grid treats them as simultaneous. The precise motion dynamics are lost, and the network suffers from the same motion blur and latency constraints that plague traditional cameras.

Moving to Learnable, Differentiable Representations

To preserve temporal structure, the conversion process itself must become a learnable part of the neural network. Instead of hardcoding how events are binned into a grid, we can define a differentiable mapping that allows gradients to flow from the final task loss all the way back to the raw event coordinates.

Daniel Gehrig et al. formalized this in End-to-End Learning of Representations for Asynchronous Event-Based Data by introducing the Event Spike Tensor (EST). Instead of assigning an event to a single temporal bin, the EST uses a differentiable kernel to linearly interpolate an event's precise timestamp across its nearest temporal grid planes. Because the interpolation is fully differentiable, the network can end-to-end optimize how it "sees" the continuous time stream.

Event Timestamp (t) ──────> [ Differentiable Interpolation ] ──────> Multi-bin Voxel Grid
                                 /                  \
                       Bin i (Weight: 1-α)     Bin i+1 (Weight: α)

A parallel approach was proposed by Stepan Tulyakov et al. in Learning an Event Sequence Embedding for Dense Event-Based Deep Stereo. They introduced a Continuous Fully-Connected (CFC) layer that processes event sequences by treating time as a continuous variable rather than a discrete index, mapping irregular event streams into dense embeddings optimized specifically for dense stereo matching.

Recurrent Memory Surfaces

While voxel grids discretize time into fixed windows, an alternative philosophy treats the pixel grid as a continuous memory surface that updates dynamically as each event arrives.

In Matrix-LSTM: a Differentiable Recurrent Surface for Asynchronous Event-Based Data, Marco Cannici et al. replaced fixed temporal binning with a grid of Long Short-Term Memory (LSTM) cells. When an event occurs at coordinates $(x,y)$, only the LSTM cell at that specific spatial location updates its internal hidden state.

Incoming Event (x, y, t, p) ───> [ LSTM Cell at (x,y) ] ───> Updated Hidden State Surface
                                                                     │
                                                       (Sampled synchronously for CNN)

This architecture provides two major intuitive advantages:

  1. Asynchronous Processing: The representation maintains the exact microsecond state of the scene, updating only where change occurs.
  2. Synchronous Readout: Whenever a downstream dense network (like a CNN) requires an input, it can sample a synchronous snapshot of the LSTM hidden states, which naturally forms a dense, feature-rich grid.

Similarly, the authors of NEST: Neural Event Stack for Event-Based Image Enhancement utilized a bidirectional ConvLSTM to encode event streams into a neural stack, capturing complex non-linear motion trajectories that are crucial for high-fidelity tasks like image deblurring and super-resolution.

Evaluating and Optimizing the Representation Space

With dozens of potential event representations available—ranging from simple surfaces to complex learnable tensors—how do we determine which representation preserves the most information without training a massive downstream network every time?

Daniel Gehrig et al. solved this bottleneck in From Chaos Comes Order: Ordering Event Representations for Object Recognition and Detection by introducing a framework based on the Gromov-Wasserstein Discrepancy (GWD). The GWD measures the topological and geometric distance between the space of raw events and the space of a dense grid representation. By finding representations that minimize this structural distortion mathematically, they bypassed the need for training entirely and discovered ERGO-12, a highly optimized representation that balances computational efficiency with task accuracy.

Taking a different optimization angle, the authors of EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision leveraged self-supervised learning to align event representations directly with traditional video frames. By forcing the event representation generator to predict underlying frame-based structures during training, the network learns to build dense representations that are natively compatible with standard vision backbones without requiring manual annotations.

Beyond the Grid: Native Asynchronous Alternatives

While converting events to dense grids remains the dominant paradigm for leveraging mature CNN architectures, some researchers argue that forcing sparse data into dense grids introduces unnecessary computational overhead.

To combat this, Rui Fan et al. proposed EventPillars: Pillar-based Efficient Representations for Event Data, which aggregates events into vertical "pillars" in the spatio-temporal space. This drastically slashes storage and computational costs by focusing processing power only on active event zones.

Taking it a step further, Piotr Wzorek et al. demonstrated in Live Demonstration: Real-time event-data processing with Graph Convolutional Neural Networks and SoC FPGA that events can be processed natively as irregular spatio-temporal point clouds using Graph Convolutional Neural Networks (GCNNs). By avoiding grid conversion entirely, their hybrid pipeline maintains true sparsity and achieves ultra-low latency processing on embedded hardware.

Whether through learnable voxel grids, recurrent memory surfaces, or native graph structures, bridging the event-to-frame gap is not about turning event cameras into standard cameras. It is about creating a mathematical translator that allows the rich, high-speed, continuous reality of neuromorphic sensors to speak the structured language of modern deep learning.

Key Foundational Work

End-to-End Learning of Representations for Asynchronous Event-Based Data (2019) — contributed 1 concept elements to this direction.
EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras (2018) — contributed 1 concept elements to this direction.
EV-SegNet: Semantic Segmentation for Event-Based Cameras (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_event_based_data_incompatibility_6b165e80,
  title = {event-based data incompatibility},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/event-based-data-incompatibility/rdr_6b165e8018df4e20967341fb6c841ab0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_event_based_data_incompatibility_6b165e80_v1,
  title = {event-based data incompatibility},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/event-based-data-incompatibility/rdr_6b165e8018df4e20967341fb6c841ab0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake