Spatiotemporal Graph Processing for Asynchronous Event Streams

25 concepts19 papers1 proposal

The Tyranny of the Frame

Traditional computer vision relies on frames: static snapshots of a scene captured at fixed intervals (e.g., 30 or 60 times per second). While intuitive, this frame-based paradigm introduces a fundamental trade-off. If an object moves too quickly between frames, it causes motion blur; if the scene is mostly static, the camera wastes energy and bandwidth repeatedly recording the exact same background pixels.

Neuromorphic (event-based) cameras offer a radical alternative. Instead of capturing full frames, each pixel operates independently and asynchronously, emitting a "spike" or event only when it detects a change in brightness. Each event is a sparse tuple containing spatial coordinates, a microsecond-level timestamp, and the direction of the brightness change.

The naive way to process this data is to accumulate events over a fixed time window into an artificial 2D frame so that standard Convolutional Neural Networks (CNNs) can read them. However, this re-introduces the very flaws event cameras were designed to fix: it causes temporal quantization, destroys microsecond-level precision, and introduces latency. To unlock the true potential of neuromorphic sensors, we must process events as they are born: asynchronously, sparsely, and continuously.

Mapping Events to Spatiotemporal Graphs

Instead of flattening time into a 2D grid, we can treat time as a literal third dimension. By plotting asynchronous events in a 3D coordinate space $(x, y, t)$, the event stream naturally forms a sparse, geometric cloud.

To process this cloud using Graph Neural Networks (GNNs), individual events are treated as nodes. Edges are then established to capture the local structure of motion. As shown by Yin Bi et al., this graph representation preserves the continuous flow and high temporal resolution of the sensor without requiring artificial frame-based grouping.

Frame-Based Binning (Information Loss)
Events:   •   •      •   •   •      •
Frames:   [  Frame 1  ]  [  Frame 2  ]  <- Time collapsed into static bins

Spatiotemporal Graph (Information Preserved)
Events:   •---•      •---•---•      •   <- Edges connect events within
               \    /                 a spatiotemporal radius
                •--•

Researchers use a few core strategies to construct these graphs:

  • Spatiotemporal Radius: Yin Bi et al. and Piotr Wzorek et al. connect vertices if their spatial and temporal distances fall within a fixed radius, capturing localized motion dynamics.
  • Topological Triangulation: Yi Zhou et al. construct graphs using Delaunay triangulation on active pixels, enforcing spatial and temporal consistency to generate sharper object masks during motion segmentation.

By keeping the data in this native 3D format, networks can extract geometric trajectories directly. For instance, Anton Mitrokhin et al. demonstrate that preserving the exact temporal evolution of a scene allows Graph Convolutional Networks (GCNs) to segment moving objects from backgrounds far more accurately than methods that collapse data into 2D projections. Similarly, Zhiyu Zhu et al. utilize GNNs to capture local geometric and temporal relationships directly within "event clouds" for robust object tracking without dense frame conversion.

Continuous-Time Modeling vs. Snapshots

The broader graph machine learning community distinguishes between two ways of handling dynamic networks: discrete snapshots (which suffer from the same binning issues as video frames) and continuous-time event streams. Comprehensive surveys by Antonio Longa et al., Alessio Gravina et al., and Zhengzhao Feng et al. highlight that continuous-time modeling avoids information loss by processing graph changes as a stream of individual, discrete events.

This continuous-time perspective is crucial for any system operating under irregular intervals. For example, in a completely different domain, Jiawei Wang et al. model unpredictable bus arrivals as a dynamic event graph to coordinate multi-agent reinforcement learning without relying on synchronized time steps.

To handle irregular intervals mathematically, a model must account for the variable passage of time between nodes. Yunhao Zhang et al. achieve this by integrating graph-based message passing directly into the intensity function of a temporal point process. The network updates node states through a combination of continuous temporal decay (representing the fading relevance of older events) and discrete, event-triggered messages.

The Mechanics of Asynchronous, Accumulation-Free Updates

If every single pixel change creates a new node, doesn't the graph grow infinitely, making computation impossibly expensive? This is where the core intuition of asynchronous processing shines.

In a standard GNN, adding a new node requires recomputing the representations for the entire graph. However, Daniel Gehrig et al. introduced AEGNN (Asynchronous Event-based Graph Neural Networks), which treats event streams as evolving graphs. Because an event camera only generates data where change occurs, a new event only affects its immediate spatiotemporal neighbors. By formulating localized update rules, the authors recompute network activations only for the nodes affected by the new event. This results in a massive 200-fold reduction in computational complexity (FLOPs) and an 8-fold reduction in latency compared to standard GNNs.

Standard GNN Recomputation:
[Node 1] \
[Node 2] -- [Global Forward Pass] -> Recomputes everything on every event
[New Node] /

Asynchronous Update (AEGNN):
[Node 1] (Unchanged)
[Node 2] -- [Local Update] -> Recomputes ONLY the immediate neighborhood
[New Node] /

To maintain this microsecond-level resolution without bottlenecking, Manon Dampfhoffer et al. propose an accumulation-free graph convolution architecture. By avoiding pooling layers and data accumulation, convolutions are applied instantly as each event arrives, breaking the traditional trade-off between prediction accuracy and processing latency. To handle the irregular spatial distribution of these streaming nodes, Dong Xiao et al. utilize spline convolutions, avoiding temporal quantization entirely.

Real-World Robustness and Hardware Realization

Operating directly on raw event graphs opens the door to ultra-low-latency applications where every millisecond counts. Dong Xiao et al. leverage this high temporal precision for real-time anomaly detection in autonomous vehicles, achieving millisecond-level response times. For continuous tracking tasks, such as high-frequency eye tracking, Nuwan Sriyantha Bandara et al. demonstrate that dynamic graph clustering can maintain accuracy during rapid movements and high-frequency transitions without requiring dense manual labels.

However, event cameras are notoriously noisy, often emitting spurious spikes due to thermal fluctuations. To prevent GNNs from overfitting to sensor artifacts, Marcin Kowalczyk et al. developed a controlled noise-injection training methodology, allowing graph convolutional networks to better distinguish true structural signals from noise.

The ultimate validation of this paradigm is its deployment on edge hardware. Because event graphs are sparse and processed incrementally, they map naturally to embedded systems. Piotr Wzorek et al. demonstrated a hardware-aware GCNN (EFGCN) running on a System-on-Chip (SoC) FPGA. By exploiting temporal sparsity through a hybrid asynchronous-synchronous pipeline, the system achieves high-accuracy object classification with latencies as low as 4.39 ms, proving that spatiotemporal graph processing is a viable path for real-time, low-power neuromorphic vision.

Key Foundational Work

Learning Visual Motion Segmentation Using Event Surfaces (2020) — contributed 2 concept elements to this direction.
Graph-Based Object Classification for Neuromorphic Vision Sensing (2019) — contributed 1 concept elements to this direction.
Real-time Prediction for Fine-grained Air Quality Monitoring System with Asynchronous Sensing (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatiotemporal_graph_representation_1745ced7,
  title = {spatiotemporal graph representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-graph-representation/rdr_1745ced758184b9bb24de17e7a5da4bd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatiotemporal_graph_representation_1745ced7_v1,
  title = {spatiotemporal graph representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatiotemporal-graph-representation/rdr_1745ced758184b9bb24de17e7a5da4bd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake