Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras

Jingkai Sun, Qiang Zhang, Jiaxu Wang, Jiahang Cao, Hao Cheng, Renjing Xu

Introduction

Dynamic Vision Sensors (DVS), commonly known as event cameras, represent a paradigm shift in visual sensing. Unlike traditional cameras that capture full frames at fixed intervals, event cameras function similarly to the human retina, asynchronously reporting pixel-level brightness changes (events) as they occur. Each event is recorded as a tuple $(x, y, t, p)$, providing microsecond-level temporal precision and high dynamic range.

Despite these advantages, the asynchronous and sparse nature of event data poses a challenge for standard computer vision models designed for dense 2D grids. Conventional approaches often collapse event streams into 2D "time-surfaces" or frames, a process that inevitably discards critical temporal dynamics and increases sensitivity to sensor noise. This paper introduces the Event Masked Autoencoder (Event MAE), a framework that processes event data directly as 3D point clouds $(x, y, t)$, preserving the high-fidelity temporal information inherent in the sensor.

Methodology: The Event MAE Framework

The Event MAE architecture is designed to learn robust spatiotemporal representations through a self-supervised masking-and-reconstruction task. The pipeline consists of several key stages: inlier patch generation, token embedding, and a transformer-based encoder-decoder structure.

The Event MAE architecture: Event data is processed through inlier downsampling, tokenized via PointNet, and then passed through a Transformer encoder-decoder for reconstruction.

Event Inlier Patch Generation

A significant hurdle in applying point-cloud methods to event data is noise. Standard sampling techniques like Farthest Point Sampling (FPS) often inadvertently select outlier noise points. To address this, the authors developed an Event Inlier Patch Generation algorithm. This method fits a local plane to groups of events using a least-squares approach; groups that fail to fit a plane well are discarded as noise. This ensures that the patches used for training represent actual physical motion rather than sensor artifacts.

Masked Autoencoding

Following patch generation, the model employs a high masking ratio (typically 80%). A PointNet-based embedding layer converts visible patches into tokens, which are then processed by a standard Transformer encoder. The decoder's task is to reconstruct the masked patches by minimizing the Chamfer Distance (CD) loss between the predicted and original point sets.

Visual demonstration of the masking and reconstruction process for a clockwise arm rotation gesture.

Key Insights

The development of Event MAE yielded several critical insights into the nature of event-based vision:

  • Temporal Dimensionality: The authors emphasize that the time dimension ($t$) in event data is fundamentally different from spatial dimensions ($x, y$). While spatial dimensions describe physical shape, time describes sequence. This distinction is why traditional 3D point cloud sampling methods often fail on event streams.
  • Self-Supervision as Denoising: The reconstruction task acts as a powerful denoising mechanism. By learning the underlying structure of motion to fill in missing gaps, the model effectively learns to distinguish signal from noise.
  • Cross-Domain Transferability: Surprisingly, pre-training the model on ShapeNet—a dataset of static 3D objects like chairs and tables—significantly boosted performance on dynamic event data. This suggests that the local geometric features of 3D shapes share fundamental similarities with the spatiotemporal "shapes" formed by moving objects in event streams.

Experimental Results

The Event MAE was evaluated against several benchmarks, consistently achieving state-of-the-art (SOTA) performance:

  • DVS128-Gesture: The model achieved a peak accuracy of 98.54%, outperforming previous leading models such as EvT and CNN+LSTM architectures.
  • SL-Animals-DVS: On the S3 set (sign language signs), the model reached 88.23% accuracy, surpassing both frame-based and spiking neural network (SNN) approaches.
  • DVS Action: The model demonstrated strong generalization, achieving 93.9% accuracy without extensive fine-tuning.

Ablation studies further validated the "Inlier" sampling method, which reduced reconstruction loss by nearly four times compared to standard FPS, proving its efficacy in handling the unique noise profile of DVS sensors.

Conclusion and Future Work

Event MAE demonstrates that treating event data as a 3D point cloud is a highly effective strategy for action recognition. By combining specialized denoising with self-supervised learning, the model captures the intricate temporal dynamics of motion that frame-based methods miss.

While the model performs exceptionally well in standard conditions, challenges remain in extremely high-noise environments, such as those with intense direct sunlight. Future research will focus on improving the robustness of patch generation and exploring multi-modal fusion to create backbones capable of processing event data alongside traditional video and LIDAR point clouds.

Summary updated

BibTeX
@inproceedings{sun2025event,
  author = {Jingkai Sun and Qiang Zhang and Jiaxu Wang and Jiahang Cao and Hao Cheng and Renjing Xu},
  title = {{Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888760},
}
Figure