Abstract
Most successful computer vision models transform low-level features, such as Gabor filter responses, into richer representations of intermediate or mid-level complexity for downstream visual tasks. These mid-level representations have not been explored for event cameras, although it is especially relevant to the visually sparse and often disjoint spatial information in the event stream. By making use of locally consistent intermediate representations, termed as superevents, numerous visual tasks ranging from semantic segmentation, visual tracking, depth estimation shall benefit. In essence, superevents are perceptually consistent local units that delineate parts of an object in a scene. Inspired by recent deep learning architectures, we present a novel method that employs lifetime augmentation for obtaining an event stream representation that is fed to a fully convolutional network to extract superevents. Our qualitative and quantitative experimental results on several sequences of a benchmark dataset highlights the significant potential for event-based downstream applications.
Introduction
Event cameras, or Dynamic Vision Sensors (DVS), represent a paradigm shift in computer vision. Unlike traditional cameras that capture frames at fixed intervals, DVS pixels report brightness changes asynchronously. This results in ultra-low latency, high dynamic range, and minimal power consumption. However, the resulting data is sparse and spatially disjointed, making it difficult for standard deep learning models—which are designed for dense, regular grids—to process effectively.
In traditional computer vision, superpixels serve as a vital mid-level representation, grouping pixels with similar characteristics to simplify scenes for tasks like object tracking or segmentation. This paper introduces "superevents," a robust equivalent for event cameras that delineates parts of objects directly from the asynchronous event stream.
The Superevent Pipeline
The authors propose a multi-stage pipeline to transform raw, noisy event data into structured, perceptually consistent segments.
1. SOFEA-BEI Pipeline
The Scene-adaptive Noise Rejection stage uses an estimated upper bound of optical flow distribution to filter out events with extreme speeds, which typically represent sensor noise. Following this, the Extended Lifetime Augmentation process ensures edges are well-defined. To prevent these edges from becoming overly thick or blurry, a Lifetime Reset mechanism is applied, thinning the edges to maintain a crisp, ideally 1-pixel thick representation.
2. Scene-adaptive Rendering
Standard rendering at fixed frame rates (e.g., 30fps) often fails to capture the dynamics of fast-moving scenes or produces redundant data for slow ones. The authors propose a motion-normalized interval where rendering is based on the expected number of events and scene complexity. This ensures the resulting Binary Event Image (BEI) remains consistent regardless of camera or object velocity.
3. FCN Superevent Extraction
The crisp BEIs are fed into a Fully Convolutional Network (FCN). This network assigns each pixel to one of nine local superpixels, creating a dense assignment map that groups events into the final "superevents."
Key Results
The researchers evaluated their method using the Event Camera Dataset, testing on diverse sequences including slider_depth, shapes_translation, and urban environments.
As shown in the figure above, the SOFEA-BEI method (b) produces significantly more detailed and crisp scene representations compared to standard methods (c). It successfully captures fine details, such as the concentric rings of a dartboard and the sharp edges of geometric shapes. The final superevent segmentation (d) demonstrates the ability to group these sparse events into meaningful local units.
Quantitatively, the SOFEA-BEI-FCN approach outperformed traditional hand-crafted methods like SLIC when applied to event data, measured by Boundary Precision and Recall. While an "intensity gap" remains—where standard APS frames (a) provide richer texture information—the event-based approach proves superior in high-speed or high-dynamic-range scenarios where traditional cameras suffer from motion blur or saturation.
Conclusion and Future Work
This work demonstrates that "superevents" provide a necessary mid-level stepping stone for neuromorphic vision. By respecting the unique physics of event cameras through motion normalization and lifetime reset mechanisms, the authors enable more effective downstream processing. Future research will focus on moving toward completely asynchronous, graph-based processing and assigning actual semantic labels (e.g., "pedestrian" or "vehicle") to these superevent groupings.
Summary updated