ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks

Yihan Lin, Wei Ding, Shaohua Qiang, Lei Deng, Guoqi LiarXiv 2021 · DOI · OpenReview

TL;DR

Researchers have introduced ES-ImageNet, a massive event-stream dataset derived from the popular ImageNet dataset, to address the critical need for large-scale benchmarks for Spiking Neural Networks (SNNs). This new dataset, generated efficiently using a novel algorithm called ODG, enables more robust training and evaluation of SNNs for neuromorphic vision tasks.

What Problem Does This Paper Address?

Spiking Neural Networks (SNNs) are a promising class of neural networks inspired by biological brains, offering potential advantages in energy efficiency and spatiotemporal processing. However, their advancement has been hampered by a lack of large-scale, high-quality datasets specifically designed for event-stream data. Existing event-stream datasets are often small and costly to acquire, making it difficult to train and validate complex SNN models. While static image datasets can be converted, existing methods often introduce significant data redundancy, negating the efficiency benefits of SNNs. This paper aims to bridge this gap by creating a large-scale, software-generated event-stream dataset that is more efficient and suitable for SNN research.

Methods

The core of this work lies in two key algorithmic contributions:

  • Omnidirectional Discrete Gradient (ODG): This novel algorithm converts static RGB images into event streams. It simulates image motion by generating local value changes based on discrete gradient information in different directions. This approach is designed to be fast and produce less data redundancy compared to previous methods. The algorithm leverages the Value (V) channel of the HSV color space as a proxy for light intensity.
  • Edge-Integral Reconstruction: To evaluate the quality of the generated event streams and enable visualization, an Edge-Integral method is used to reconstruct grayscale images from the event data. This process is crucial for understanding information loss and validating the conversion process.

The authors also detail their process for selecting hyper-parameters for the ODG algorithm, including trajectory, threshold, and time steps, through preparatory experiments and analysis of 2D-entropy.

Datasets

  • ES-ImageNet: This is the primary contribution of the paper. It's an event-stream (ES) version of the ILSVRC2012 (ImageNet) dataset, containing approximately 1.3 million samples across 1000 categories. It is claimed to be the largest ES-dataset for object classification to date.
  • ILSVRC2012: The original static image dataset used as the source for generating ES-ImageNet.
  • Existing ES-datasets: The paper compares ES-ImageNet to several existing event-stream datasets like N-MNIST, N-Caltech101, DVS-Gesture, and CIFAR10-DVS, highlighting ES-ImageNet's significantly larger scale.

Key Insights and Ideas

  • Software Generation is Key for Scale: Creating large-scale event-stream datasets from real neuromorphic cameras is prohibitively time-consuming and expensive. Software-based conversion, when optimized, offers a viable path to large datasets.
  • Data Redundancy is a Major Bottleneck: Previous methods for converting static images to event streams often introduced significant redundancy, undermining the efficiency advantages of SNNs. The ODG algorithm specifically targets this issue.
  • SNNs Thrive on Sparse, Temporal Data: The paper emphasizes that SNNs are naturally suited for processing the sparse and temporal nature of event-stream data, making ES-datasets ideal for their training and evaluation.
  • ES-ImageNet as a New Benchmark: The sheer scale and diversity of ES-ImageNet make it a challenging and valuable benchmark for pushing the boundaries of SNN research and neuromorphic vision.

Key Results

  • Dataset Creation: Successfully generated ES-ImageNet with ~1.3 million samples in 1000 categories from ILSVRC2012.
  • Performance Benchmarking:
    • ANNs: 2D-CNN achieved 43.14% accuracy on ResNet-18 and 45.38% on ResNet-34. 3D-CNN achieved 41.03% on ResNet-18 and 42.74% on ResNet-34.
  • SNNs: LIF-SNN (baseline) achieved 39.89% on ResNet-18 and 43.42% on ResNet-34. LIAF-SNN achieved the highest accuracy among all tested models: 42.54% on ResNet-18 and 47.47% on ResNet-34.
  • Energy Efficiency: SNNs, particularly LIAF-SNN, demonstrated significant energy advantages over CNNs, with LIF-SNN consuming about half the energy of CNNs.
  • Information Analysis: 2D-Entropy analysis suggests that ES-ImageNet retains a substantial amount of information, comparable to compressed RGB images, making it suitable for classification.

Limitations and Future Work

  • Artificial Motion: The event streams are generated by simulating image motion, which differs from real-world dynamic scenes captured by DVS cameras.
  • Reduced Randomness: To compress data and extract information, the randomness of generation was reduced, potentially losing some characteristics of DVS recordings.
  • Information Loss Metrics: 2D-Entropy measures information quantity but not necessarily effective information. Reconstruction processes can also introduce information loss.
  • Limited Hardware/Algorithm Scope: The experiments were limited to ResNet-18 and ResNet-34 architectures due to hardware and algorithmic constraints.
  • Future Work: The authors suggest exploring data augmentation techniques (e.g., random order, dynamic time length, clipping) and pre-training for SNNs on ES-ImageNet. They also hope for more research to test larger and more advanced SNN models and other neuromorphic algorithms on this dataset.

Summary updated

Abstract

With event-driven algorithms, especially the spiking neural networks (SNNs), achieving continuous improvement in neuromorphic vision processing, a more challenging event-stream-dataset is urgently needed. However, it is well known that creating an ES-dataset is a time-consuming and costly task with neuromorphic cameras like dynamic vision sensors (DVS). In this work, we propose a fast and effective algorithm termed Omnidirectional Discrete Gradient (ODG) to convert the popular computer vision dataset ILSVRC2012 into its event-stream (ES) version, generating about 1,300,000 frame-based images into ES-samples in 1000 categories. In this way, we propose an ES-dataset called ES-ImageNet, which is dozens of times larger than other neuromorphic classification datasets at present and completely generated by the software. The ODG algorithm implements an image motion to generate local value changes with discrete gradient information in different directions, providing a low-cost and high-speed way for converting frame-based images into event streams, along with Edge-Integral to reconstruct the high-quality images from event streams. Furthermore, we analyze the statistics of the ES-ImageNet in multiple ways, and a performance benchmark of the dataset is also provided using both famous deep neural network algorithms and spiking neural network algorithms. We believe that this work shall provide a new large-scale benchmark dataset for SNNs and neuromorphic vision.
BibTeX
@misc{lin2021es,
  author = {Yihan Lin and Wei Ding and Shaohua Qiang and Lei Deng and Guoqi Li},
  title = {{ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks}},
  year = {2021},
  eprint = {2110.12211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12211},
}
Figure