Traditional computer vision is built on a synchronous paradigm: cameras capture full grids of pixels at fixed time intervals, producing a sequence of dense frames. Event cameras (or Dynamic Vision Sensors) completely upend this model. Instead of capturing frames, each pixel operates independently and asynchronously, emitting an "event" only when it detects a localized change in log-intensity brightness. An event is a sparse, microsecond-accurate tuple: $(x, y, t, p)$, where $x$ and $y$ are spatial coordinates, $t$ is the precise timestamp, and $p \in \{-1, 1\}$ represents the polarity (brightness increase or decrease).
This asynchronous sparsity presents a fundamental impedance mismatch. Modern deep learning architectures—such as Convolutional Neural Networks (CNNs), Transformers, and Diffusion models—are highly optimized for dense, synchronous, grid-like tensors. Passing a raw, irregular stream of millions of individual events directly into a standard ResNet or Vision Transformer is computationally intractable and structurally incompatible.
To bridge this gap, researchers use spatio-temporal aggregation (or event binning). By accumulating asynchronous event streams over specified temporal windows into structured, multi-dimensional tensors, this technique preserves the high-temporal resolution of neuromorphic sensors while unlocking the power of standard computer vision backbones.
Collapsing Time: Event Frames and Histograms
The most intuitive way to build a grid from an event stream is to accumulate events over a fixed temporal window $\Delta t$ into a 2D spatial grid. By counting the number of events that occurred at each pixel $(x, y)$, we generate an event histogram. To preserve the directional context of illumination changes, positive and negative polarities are typically separated into distinct channels, creating a tensor of shape $(2, H, W)$.
As demonstrated by Ana I. Maqueda et al. in Event-Based Vision Meets Deep Learning on Steering Prediction for Self-Driving Cars and by Roshan Gopalakrishnan et al. in Classifying neuromorphic data using a deep learning framework for image classification, this simple accumulation transforms sparse spikes into static, edge-like images.
Why does this work so effectively for standard neural networks? Event cameras naturally function as high-pass spatial-temporal filters; they remain silent in uniform regions and fire rapidly along moving boundaries. Accumulating these events over a short window reconstructs the sharp outlines and profiles of moving objects. Because early layers in standard CNNs naturally learn Gabor-like edge filters, these aggregated event frames are immediately compatible with pre-trained spatial backbones.
Taking this concept further, Ziyi Wu et al. introduced EventCLIP: Adapting CLIP for Event-based Object Recognition, which maps multi-channel polarity histograms into a three-channel format matching standard RGB inputs. This structural alignment allows event data to directly leverage the massive, zero-shot visual-language knowledge embedded in pre-trained models like CLIP, bypassing the scarcity of labeled neuromorphic datasets.
Preserving Motion Dynamics: Spatio-Temporal Voxel Grids
While 2D histograms capture spatial structure, they suffer from a major limitation: they discard the sequence of events within the window $\Delta t$. If an edge moves across a pixel grid, a simple histogram shows a blurred trail of counts but cannot tell you which way or how fast the edge moved. For motion-critical tasks like optical flow estimation or tracking, preserving the arrow of time is essential.
To solve this, the temporal window is discretized into $B$ sequential bins, forming a 3D voxel grid of shape $(B, H, W)$. In Live Demonstration: Unsupervised Event-Based Learning of Optical Flow, Depth and Egomotion, Alex Zihao Zhu et al. demonstrated that this volumetric representation allows standard 2D convolutions to treat the temporal bins as input channels. The network can then compare adjacent channels to determine exactly how an edge shifted from bin $t$ to bin $t+1$, extracting precise velocity cues.
Asynchronous Events Spatio-Temporal Voxel Grid
(x, y, t, p) (B x H x W Tensor)
| +---+---+---+
| Temporal Partitioning | | | | Bin B (Latest)
+-----------------------------> +---+---+---+
| | | | ...
+---+---+---+
| | | | Bin 1 (Earliest)
+---+---+---+
However, simply assigning events to the nearest temporal bin introduces quantization noise, which destroys sub-millisecond precision. To preserve this fine-grained timing, Kenneth Chaney utilized trilinear interpolation in Learning Event-Based Height From Plane and Parallax. When an event falls between bins, its polarity is distributed proportionally to its nearest spatial and temporal neighbors. This mathematical formulation ensures that microsecond-level timing variations are smoothly encoded as continuous intensity values, keeping the representation fully differentiable and rich in motion gradients.
Encoding History: Time Surfaces
Another elegant strategy for capturing temporal dynamics without increasing the dimensionality of the tensor is the Time Surface or Surface of Active Events (SAE). Instead of counting events or creating multiple temporal bins, a time surface maintains a 2D grid where each pixel value stores a normalized timestamp of the most recent event at that coordinate.
As highlighted by Aayush Atul Verma et al. in eTraM: Event-Based Traffic Monitoring Dataset, these timestamps are typically scaled so that recent events appear bright and older events decay toward zero.
To understand why this works intuitively, imagine a bright vertical bar moving from left to right across the sensor. The pixels on the right edge of the bar will have just fired an event, so their coordinates on the time surface will contain highly recent timestamps (high values). The pixels further to the left, which the bar passed through moments ago, will contain older timestamps (lower values). This creates a smooth spatial ramp or gradient across the surface.
When a standard spatial convolution filter processes this gradient, it can immediately deduce both the direction of motion (orthogonal to the gradient contours) and the velocity (a steeper gradient implies faster motion). This makes time surfaces exceptionally compact and computationally efficient for real-time edge applications, such as robotic grasping (Bin Li et al. in Event-Based Robotic Grasping Detection With Neuromorphic Vision Sensor and Event-Grasping Dataset).
From Handcrafted to Learnable Representations
Despite their success, handcrafted representations like fixed voxel grids or time surfaces rely on human heuristics. They assume that uniform temporal binning or exponential decay is optimal for every computer vision task, which is rarely the case.
To overcome this limitation, Daniel Gehrig et al. proposed a generalized, fully differentiable framework in End-to-End Learning of Representations for Asynchronous Event-Based Data. They introduced the Event Spike Tensor (EST), which unifies handcrafted methods by treating the transformation from an asynchronous event stream to a grid-based tensor as a learnable neural network layer.
The EST passes raw events through a continuous kernel convolution and quantization step where the kernel parameters are not fixed, but are instead optimized via backpropagation alongside the downstream task network. By making the aggregation step end-to-end learnable, the model dynamically tailors the input tensor format to maximize performance for the specific task at hand—whether that means sharpening spatial boundaries for object recognition or maximizing temporal precision for high-speed tracking.
Summary
Spatio-temporal aggregation acts as the vital translator between neuromorphic hardware and deep learning backbones. Whether by collapsing time into an event frame to reuse image models, slicing time into a voxel grid to track motion sequences, or mapping recency onto a time surface to expose velocity gradients, transforming asynchronous spikes into structured tensors allows researchers to harness the ultra-low latency and high dynamic range of event cameras using the robust deep learning tools already at their disposal.