Flow-Event Autoencoder: Event Stream Object Recognition Dataset Generation with Arbitrary High Temporal Resolution

Minghai ChenAAAI 2024 · DOI

TL;DR

This paper introduces a novel "flow-event-flow" autoencoder that generates high-temporal-resolution event stream data from static images and optical flow. This method addresses the critical lack of large-scale event-based datasets for object recognition, offering a hardware-free, efficient, and generalizable solution that promises to accelerate the development of neuromorphic vision systems.

What Problem Does This Paper Address?

Event cameras, like Dynamic Vision Sensors (DVS), offer compelling advantages such as low power consumption and high temporal resolution. However, their advancement in computer vision, particularly for object recognition, is significantly hampered by the scarcity of large, benchmark datasets. Existing methods for generating event streams often suffer from limitations like low efficiency, dependence on screen refresh rates, poor temporal resolution, lack of noise modeling, or are not designed for generating from static images. This paper tackles the challenge of creating a robust and flexible method to generate event stream data from readily available image and optical flow information, enabling the creation of larger and more diverse datasets for training and evaluating event-based vision models.

Methods

The core of this work is a Vision Transformer (ViT) encoder-decoder architecture, termed "flow-event-flow." This model learns to generate event streams by taking an image and pre-defined motion trajectory (represented by optical flow) as input. The ViT's self-attention mechanism is chosen for its ability to capture global information.

The training process involves a two-stage approach:

  1. Pre-training a Decoder: A separate decoder network is first trained to reconstruct optical flow from event streams. This leverages existing successful methods in event-based optical flow estimation.
  2. Training the Encoder-Decoder: The main encoder-decoder then learns to generate event streams. The training objective includes two losses:
  3. An L1 loss comparing the generated events to ground truth events derived from the optical flow.
  4. A loss based on the reconstructed optical flow from the generated events using the pre-trained decoder. This forces the encoder to produce realistic events that can be used to accurately estimate optical flow, ensuring the generated events are meaningful and aligned with motion.

This approach allows for the generation of event streams at arbitrary temporal resolutions by tuning the interpolation window used for optical flow extraction.

Datasets

The authors propose to construct a training dataset by adapting the setup used in Orchard et al. (2015) with the ImageNet dataset. Images from ImageNet are printed on non-reflective paper to mitigate screen refresh rate issues. Ground truth optical flow is extracted from the motion trajectory of a simulated saccadic camera movement.

For evaluation, the generated event streams will be used to train models that are then evaluated on:

  • N-imageNet-1k: A dataset generated using their proposed method.
  • ImageNet-1k: The original image dataset used for generation.
  • N-Caltech101: A smaller, existing event-based object recognition benchmark.
  • N-imageNet-1k (mechanically generated): For comparison with hardware-based generation.

Key Insights and Ideas

  • Bridging the Dataset Gap: The primary insight is that generating event streams from static images and optical flow is a viable and efficient strategy to overcome the dataset scarcity in event-based vision.
  • Leveraging Pre-trained Decoders: Using a pre-trained event-to-optical-flow decoder as part of the training loss is a clever way to ensure the generated events are not just visually plausible but also functionally useful for downstream tasks like motion estimation.
  • Arbitrary Temporal Resolution: The method's ability to generate event streams at any desired temporal resolution is a significant advantage, allowing for fine-grained analysis and training.
  • Hardware-Free Generation: Eliminating the need for specialized hardware setups after the initial training phase makes the generation process highly scalable and accessible.

Key Results

The paper outlines an evaluation strategy where their generated event streams are used to train object recognition models. They expect their method to:

  • Out-perform other simulation methods like ESIM-vid2e.
  • Achieve comparable or even superior results to hardware-based generation methods like the one used for N-Caltech101.

Specific quantitative results are not detailed in the provided text, as the paper focuses on the proposed method and evaluation strategy.

Limitations and Future Work

Limitations: * The initial training dataset construction requires printing images, which, while avoiding screen refresh issues, is still a physical process. * The method relies on pre-defined saccades motion trajectories, which might not perfectly capture all real-world motion dynamics.

Future Work: The authors envision their method as a strategy to enrich event-based vision datasets, enabling large-scale pretraining and transfer learning. This can fuel the development of neuromorphic vision for applications like self-driving, high-speed drones, and robotics. The proposed method is presented as a foundational step towards a more mature event-based vision paradigm.

Summary updated

BibTeX
@inproceedings{chen2024flow,
  author = {Minghai Chen},
  title = {{Flow-Event Autoencoder: Event Stream Object Recognition Dataset Generation with Arbitrary High Temporal Resolution}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30545},
}
Figure