Traditional computer vision is built on frames—static snapshots of a scene captured at regular intervals. Event cameras (or Dynamic Vision Sensors) completely break this paradigm. Instead of capturing full images, they record an asynchronous stream of per-pixel brightness changes, outputting a continuous sequence of events characterized by their spatial coordinates, precise microsecond timestamps, and binary polarity (signifying an increase or decrease in light).
Historically, researchers forced this elegant, sparse data into traditional deep learning pipelines by accumulating events over fixed time windows into artificial 2D frames or dense 3D voxel grids. However, this conversion introduces a fundamental conflict: quantization destroys the microsecond-level temporal precision of the sensor, introduces motion blur, and inflates computational overhead by turning sparse data into dense tensors.
To resolve this, a powerful research direction has emerged: treating the asynchronous event stream directly as a spatiotemporal point cloud (or "event cloud"). By treating time as a geometric dimension alongside space, researchers can leverage the rich toolkit of 3D computer vision to process neuromorphic data natively, preserving both its extreme temporal resolution and its inherent sparsity.
The Geometric Intuition: Time as Space
When an object moves in front of an event camera, it leaves behind a continuous trail of triggered pixels. If you stack these events along a temporal axis, the stream forms a distinct geometric manifold in a three-dimensional $(x, y, t)$ space.
As noted by Zhiyu Zhu et al. in Learning Graph-embedded Key-event Back-tracing for Object Tracking in Event Clouds, directly consuming these raw spatiotemporal event clouds preserves the fine-grained trajectories of moving objects without the information loss or motion blur typically introduced by frame-based accumulation.
This geometric framing allows researchers to apply architectures originally designed for spatial 3D point clouds (like LiDAR scans) directly to temporal data. For example, Jiaan Chen et al. demonstrate in Efficient Human Pose Estimation via 3D Event Point Cloud that mapping microsecond timestamps to a spatial dimension allows 3D deep learning architectures—such as PointNet and Point Transformers—to capture the exact geometric shape of human motion with exceptionally low latency.
Frame-Based Accumulation (Information Loss):
Asynchronous Events (x, y, t) ---> Quantization into 2D Grid ---> Motion Blur & Latency
Direct Event Cloud Processing (Geometry Preserving):
Asynchronous Events (x, y, t) ---> 3D/4D Spatiotemporal Point Cloud ---> Point/Graph Networks
Analytical Geometry on Event Surfaces
Treating events as geometric points benefits classical, non-learning-based estimation methods as well. Instead of optimizing for image sharpness through iterative warping, algorithms can solve geometric problems directly on the spatiotemporal manifold.
In Spatiotemporal Registration for Event-Based Visual Odometry, Daqi Liu et al. show that motion estimation can be framed as registering subsets of events using geometric distance metrics between back-projected rays. Because this approach operates directly on the point cloud rather than rendering images, its computational cost is completely independent of the camera's sensor resolution, enabling the real-time processing of millions of events per second.
For specific structural primitives, like lines, the geometry becomes even simpler. Alexander Dietsche et al. demonstrate in Powerline Tracking with Event Cameras that a moving linear feature traces out a perfect plane within the $(x, y, t)$ coordinate system. Rather than relying on appearance-based matching, the system can estimate and track lines by calculating the eigen-decomposition of the event covariance matrix, providing robust tracking even when event density drops significantly.
Architectural Paradigms for Event Clouds
Natively processing spatiotemporal point clouds requires architectures that respect the unique properties of time. As Anton Mitrokhin et al. point out in Learning Visual Motion Segmentation Using Event Surfaces, naive 3D processing can fail if it treats the time axis exactly like a standard spatial dimension, ignoring the underlying surface topology of motion. To capture these dynamics accurately, researchers have developed several specialized neural network paradigms:
Graph and Attention Networks
To model the continuous nature of event surfaces, Anton Mitrokhin et al. utilize Graph Convolutional Networks (GCNs) to segment moving objects by treating sparse event data as a 3D graph in $(x, y, t)$ space. When the complexity of the interaction increases, the dimensionality of the cloud can be expanded. For instance, Christen Millerdurai et al. develop a framework in 3D Pose Estimation of Two Interacting Hands from a Monocular Event Camera that aggregates events into short windows represented as 5D point clouds (spatial coordinates, timestamps, and polarity). This high-dimensional representation allows attention mechanisms to track intricate, high-speed hand interactions and occlusions without losing spatial structure.
Lightweight Edge Architectures
For low-power robotics, event cloud networks must remain highly efficient. Hongwei Ren et al. address this in SpikePoint: An Efficient Point-based Spiking Neural Network for Event Cameras Action Recognition by combining point-based sampling and clustering with Spiking Neural Networks (SNNs), maintaining the raw geometric structure while drastically reducing computational overhead. In a subsequent architecture, Scalable Event Cloud Network for Event-based Classification, the authors shift feature extraction into the frequency domain via Fourier Transforms, bypassing heavy voxelization steps entirely to achieve scalable, lightweight classification.
Spatial Relocalization and Cross-Modal Registration
Point-based representations also excel at spatial localization tasks. In A Simple and Effective Point-based Network for Event Camera 6-DOFs Pose Relocalization, Hongwei Ren et al. introduce PEPNet, demonstrating that a point-based $(x, y, t)$ representation allows a tiny network to accurately capture motion trajectories for 6-DoF camera relocalization using a fraction of the parameters required by frame-based models. To bridge the gap between 2D event sensors and 3D spatial models, Xiuhong Lin et al. design E2PNet: Event to Point Cloud Registration with Spatio-Temporal Representation Learning, which extracts geometric features directly from spatiotemporal event coordinates to register them directly against standard 3D spatial point clouds.
Hybrid Projections and Generative Reconstruction
While direct point processing is highly efficient, certain downstream tasks (like dense object detection) benefit from structured intermediate representations.
To combine the best of both worlds, Dongsheng Wang et al. propose Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation. Their method, EventPillars, treats events as points in a 3D spatiotemporal space but groups them into vertical pillars. A convolutional encoder then learns how to project these pillars into a dense pseudo-image, replacing hand-crafted event histograms with an optimized, learnable representation.
Hybrid Pillar Projection:
Raw Events (x, y, t) ---> Group into Spatiotemporal Pillars ---> Learnable Encoder ---> Dense Pseudo-Image
When event streams suffer from data loss due to sensor bandwidth limits or extreme conditions, the geometric point cloud perspective also provides a powerful framework for data restoration. In An event-oriented diffusion-refinement method for sparse events completion, Bo Zhang et al. treat event completion as a 3D geometric reconstruction task rather than a 2D image inpainting problem. By utilizing a coarse-to-fine generative diffusion model directly on the spatiotemporal point cloud, their system rebuilds missing sensor data with microsecond precision.
Application Domains
The shift toward direct spatiotemporal point cloud processing has unlocked state-of-the-art performance across a wide variety of high-speed, low-power computer vision applications:
- Eye Tracking: In a comprehensive survey, Zuowen Wang et al. (Event-Based Eye Tracking. AIS 2024 Challenge Survey) highlight how top-performing teams leverage point-based networks, sparse convolutions, and state-space models on raw event streams to achieve sub-millisecond latency and high precision for next-generation AR/VR headsets.
- In-the-Wild Tracking: For monitoring fast-moving objects, Gabriele Magrini et al. introduce the EV-Flying dataset in EV-Flying: an Event-based Dataset for In-The-Wild Recognition of Flying Objects, validating 4D point cloud representations (incorporating spatial coordinates, precise timestamps, and polarity changes) to track objects where traditional cameras fail due to motion blur.
- Action Recognition: For human behavior analysis, Xing Li 0005 et al. demonstrate in PointMapNet: Point Cloud Feature Map Network for 3D Human Action Recognition that directly processing point cloud sequences bypasses computationally intensive voxelization steps, offering a highly efficient alternative for extracting clean spatiotemporal dynamics.
By discarding the artificial constraint of frames and treating time as a native coordinate, spatiotemporal point cloud representations preserve the core strengths of neuromorphic sensors: speed, sparsity, and mathematical precision.
Key Foundational Work
Spatiotemporal Registration for Event-Based Visual Odometry (2021) — contributed 1 concept elements to this direction.
Learning Visual Motion Segmentation Using Event Surfaces (2020) — contributed 1 concept elements to this direction.
Powerline Tracking with Event Cameras (2021) — contributed 1 concept elements to this direction.