Traditional computer vision operates like a flipbook, capturing the world as a sequence of static, synchronous frames sampled at fixed intervals. This paradigm introduces inherent limitations: fast-moving objects become motion-blurred, and critical changes occurring between frames are lost entirely.
Neuromorphic event cameras offer a radical alternative. Instead of capturing full images at a fixed rate, each pixel operates independently and asynchronously, emitting a digital "event" (or spike) the exact microsecond it detects a change in log-brightness. This results in a continuous, highly sparse data stream with microsecond-level temporal resolution.
However, standard deep learning architectures—such as Convolutional Neural Networks (CNNs) and Transformers—are fundamentally designed for dense, grid-like tensors. To bridge this architectural gap, researchers traditionally aggregate asynchronous events over a fixed time window into a single 2D frame or a 3D voxel grid. This conversion step creates a severe representation bottleneck, discarding the primary benefits of neuromorphic hardware.
Why Frame Conversion Destroys Information
When a continuous stream of events is forced into a discrete frame or voxel grid, three main types of degradation occur:
- Temporal Aliasing and Overwriting: If a pixel experiences multiple brightness changes within a single integration window, older events are either overwritten or collapsed into a simple sum. As noted by Anh Nguyen et al. in Real-Time 6DOF Pose Relocalization for Event Cameras With Stacked Spatial LSTM Networks., this overwriting effect removes the precise sequence of brightness changes, preventing downstream models from leveraging high-frequency temporal dynamics.
- Artificial Latency: Accumulating events over a time window forces the system to wait for the window to close before processing can begin. This introduces an artificial delay, neutralizing the microsecond-level latency of the underlying sensor.
- Loss of Sparsity: Event streams are naturally sparse, only generating data where motion or illumination changes occur. Converting them into dense frames forces networks to waste memory and computational cycles processing empty spatial regions, as highlighted by Daniel Gehrig et al. in AEGNN: Asynchronous Event-based Graph Neural Networks.
To overcome this bottleneck, recent research focuses on representations that treat time not as a discrete bin index, but as a continuous coordinate or a native dimension.
Preserving Trajectories via Graphs and Point Clouds
One powerful intuition is to treat the event stream as a geometric structure in continuous $(x, y, t)$ space. By preserving the exact timestamp $t$ of each event, models can track continuous motion trajectories without quantization error.
Frame-Based Binning (Lossy):
Events: --*---*--**---*---> (Continuous Time)
Binned: [ 3 events ] [ 2 ] -> Collapsed into static snapshots
Spatiotemporal Graph (Preserved):
Events become nodes connected to spatial/temporal neighbors.
Time (t) remains an explicit, continuous coordinate.
Instead of flattening events into 2D pseudo-frames, Anton Mitrokhin et al. demonstrate in Learning Visual Motion Segmentation Using Event Surfaces that treating sparse event data as a 3D graph preserves the microsecond-level trajectories required to segment fast-moving objects from the background. Similarly, Yin Bi et al. introduce a framework in Graph-Based Object Classification for Neuromorphic Vision Sensing that processes spikes as nodes in a Residual Graph Convolutional Neural Network (RG-CNN), maintaining high temporal precision while significantly reducing memory consumption compared to frame-based alternatives.
This geometric framing can also be extended to 3D point clouds. In Efficient Human Pose Estimation via 3D Event Point Cloud, Jiaan Chen et al. bypass the frame-based detour entirely by treating the event stream as a "rasterized event point cloud." By utilizing 3D deep learning backbones like PointNet and Point Transformers, the authors achieve real-time tracking with minimal latency, demonstrating that preserving the raw spatiotemporal distribution of events is key to handling high-speed human motion.
Native Asynchronous Processing with Spiking Networks
If the goal is to fully exploit the low-power and low-latency characteristics of neuromorphic sensors, the processing architecture should match the asynchronous nature of the hardware. Spiking Neural Networks (SNNs) achieve this by maintaining an internal state (membrane potential) that updates only when a new event arrives.
In Event-Based Angular Velocity Regression with Spiking Networks, Davide Scaramuzza et al. show that SNNs can successfully perform continuous-time regression tasks, such as predicting camera rotation, without forcing data into a rigid, discrete-time format that reintroduces motion blur. This native processing paradigm is highly effective for resource-constrained edge applications. For instance, Pietro Bonazzi et al. leverage this synergy in Retina : Low-Power Eye Tracking with Event Camera and Spiking Hardware, deploying an SNN directly onto specialized neuromorphic hardware to achieve sub-10ms pupil tracking at an ultra-low power budget of 5mW.
End-to-End Learnable Representations
When interfacing with traditional visual architectures or large-scale models (such as CLIP) is unavoidable, researchers have replaced handcrafted, lossy quantization heuristics with differentiable, learnable transformations.
The core idea is to let the neural network optimize the event-to-tensor mapping based on the specific requirements of the downstream task. Daniel Gehrig et al. pioneer this approach in End-to-End Learning of Representations for Asynchronous Event-Based Data by introducing the Event Spike Tensor (EST). The EST framework uses a differentiable kernel to grid events into a tensor, allowing the network to retain critical temporal resolution and polarity details that manual heuristics typically discard.
For complex image restoration tasks like deblurring and super-resolution, Minggui Teng proposes NEST: Neural Event Stack for Event-Based Image Enhancement. Instead of relying on static voxel grids, NEST employs a bidirectional ConvLSTM to dynamically encode the event stream, capturing non-linear motion dynamics and preserving the fine-grained temporal information necessary to reconstruct high-quality video from blurred frames.