Abstract
Neuromorphic Vision sensors have improved greatly since the first silicon retina was presented almost three decades ago. They have recently matured to the point where they are commercially available and can be operated by laymen. However, despite improved availability of sensors, there remains a lack of good datasets, while algorithms for processing spike-based visual data are still in their infancy. On the other hand, frame-based computer vision algorithms are far more mature, thanks in part to widely accepted datasets which allow direct comparison between algorithms and encourage competition. We are presented with a unique opportunity to shape the development of Neuromorphic Vision benchmarks and challenges by leveraging what has been learnt from the use of datasets in frame-based computer vision. Taking advantage of this opportunity, in this paper we review the role that benchmarks and challenges have played in the advancement of frame-based computer vision, and suggest guidelines for the creation of Neuromorphic Vision benchmarks and challenges. We also discuss the unique challenges faced when benchmarking Neuromorphic Vision algorithms, particularly when attempting to provide direct comparison with frame-based computer vision.
The Data Gap in Neuromorphic Vision
Neuromorphic vision (NV) sensors, or event-based cameras, operate on a fundamentally different principle than traditional frame-based cameras. Instead of capturing global snapshots at fixed intervals, they capture asynchronous "events" triggered by per-pixel changes in light intensity. This paradigm offers microsecond temporal resolution and significant power efficiency, but it creates a compatibility gap with existing computer vision (CV) infrastructure.
The field currently faces a "chicken and egg" dilemma: the development of sophisticated spike-based algorithms is hindered by a lack of large-scale, annotated datasets, while the creation of such datasets is slowed by the difficulty of processing and labeling event-based data. Unlike traditional CV, where static scenes are easily captured, NV sensors require motion—either from the scene or the sensor itself—to generate data.
Lessons from the Evolution of Computer Vision
By analyzing the history of CV benchmarks—from the early days of MNIST and Caltech-101 to the transformative impact of ImageNet—we can identify a blueprint for NV development. The "deep learning revolution" was not just a result of better algorithms, but of the availability of massive, standardized datasets that allowed for objective comparison and competition.
Current NV datasets generally fall into three categories: raw unannotated recordings, small task-specific datasets that are often "solved" too quickly to drive long-term progress, and converted datasets where NV sensors record standard CV datasets (like MNIST) displayed on screens. To move forward, the community must transition toward more complex, real-world benchmarks.
Strategic Recommendations for Benchmarking
To catalyze growth, the NV community should adopt several key strategies:
- Embrace Active Perception: In biology, microsaccades prevent retinal images from fading. NV sensors should similarly utilize sensor motion to generate data from static environments, bridging the gap with traditional object recognition.
- Focus on Natural Advantages: NV should not merely compete with CV on static image tasks. Instead, benchmarks should focus on high-speed motion, low-latency requirements, and power-constrained environments like mobile robotics or retinal prostheses.
- Hybrid Data Collection: To ensure a "fair fight" and facilitate cross-domain learning, new datasets should utilize hybrid sensors that capture both frames and events simultaneously.
- Open Source and Reproducibility: The maturity of CV was driven by shared code and data. NV researchers must move away from proprietary formats and toward reproducible, public repositories.
The Annotation Challenge
One of the most significant hurdles in NV benchmarking is the complexity of ground-truthing. In traditional CV, a bounding box on a 2D image is sufficient. In NV, data is a continuous stream of millions of points in a 3D spatio-temporal volume (XY + Time).
Annotating this stream is a significant challenge. While simulators offer a potential solution for generating ground truth, they often fail to capture the complex noise profiles of real silicon retinas. Recognizing dataset creation as a primary research contribution is essential for overcoming these technical barriers and eventually creating a "Neuromorphic ImageNet."
Summary updated