Imaging from temporal data via spiking convolutional neural networks

Paul Kirkland, Valentin Kapitány, Ashley Lyons, John J. Soraghan, Alex Turpin, Daniele Faccio, Gaetano Di Caterina2020 · DOI

TL;DR

This paper introduces Spike-SPI, a novel Spiking Convolutional Neural Network (SCNN) for 3D depth imaging using only time-of-flight (ToF) data from a single-pixel detector. Spike-SPI achieves state-of-the-art spatial accuracy and significantly improves signal-to-noise ratio compared to traditional ANNs, while also offering reduced computational overhead.

What Problem Does This Paper Address?

Traditional 3D imaging methods often rely on flood illumination and multi-pixel detectors, which can be bulky and computationally intensive. Alternatively, single-pixel scanning methods offer advantages in size and frame rate but struggle with reconstructing spatial information from time-of-flight (ToF) data alone. This is because a single photon's arrival time only tells us its distance from the sensor, not its precise location in 3D space, leading to an ill-posed inverse problem. This paper tackles the challenge of reconstructing accurate 3D depth maps from ToF data acquired by a single-pixel sensor, aiming to combine the benefits of high speed and compact hardware.

Methods

The core of the proposed solution is a Spiking Convolutional Neural Network (SCNN) named Spike-SPI. This network combines several key components:

  • Convolutional Neural Networks (CNNs): Utilized for their powerful feature extraction capabilities, enabling the network to learn subtle patterns within the 1D temporal histograms of photon arrival times.
  • Variational Encoder-Decoder Structure: This architecture is employed to transform the 1D temporal data into a 3D depth map, inferring spatial features from the temporal information.
  • Spiking Neural Networks (SNNs): The network leverages the asynchronous processing and inherent sparsity of SNNs. This allows for more efficient computation and potentially lower latency compared to traditional Artificial Neural Networks (ANNs).

The approach involves:

  1. Illuminating a scene with a pulsed laser.
  2. Detecting back-scattered photons with a single-point detector (SPAD) and recording their arrival times using Time Correlated Single Photon Counting (TCSPC).
  3. Transforming these arrival times into a temporal histogram.
  4. Feeding this histogram into the Spike-SPI SCNN to reconstruct a 3D depth map.

The SCNN is trained using synthetic data that simulates the ToF acquisition process.

Datasets

The paper utilizes synthetic data generated to simulate the imaging setup. This synthetic dataset includes humanoid silhouettes in various poses against a background of objects, within a 20m³ 3D environment. The ground truth 3D depth information is encoded in the color of the scene. The training and validation data were split into four experiments based on different photon counts (1000 and 9500) and Instrument Response Functions (IRFs) (20ps and 100ps) to test the network's robustness.

Key Insights and Ideas

  • Bridging the Gap: The research demonstrates that it's possible to reconstruct detailed 3D spatial information from purely temporal data acquired by a single-pixel sensor, overcoming the inherent ill-posed nature of the problem.
  • Spiking for Efficiency: The integration of spiking neurons offers a significant advantage in terms of computational efficiency and potentially faster processing, leveraging the sparse and asynchronous nature of photon detection.
  • CNNs for Temporal Patterns: The success of CNNs in extracting spatial features is effectively repurposed to extract meaningful temporal features from the photon arrival histograms.
  • Encoder-Decoder for Reconstruction: The encoder-decoder architecture proves adept at translating the extracted temporal features into a coherent 3D depth map.

Key Results

Spike-SPI demonstrated superior performance compared to a standard ANN model in several key metrics:

  • Spatial Accuracy: Spike-SPI achieved a 15% increase in spatial accuracy, as measured by the Intersection over Union (IoU) for objects in the scene.
  • Signal-to-Noise Ratio (RSNR): The network showed a 100% increase in object reconstruction signal-to-noise ratio, improving from approximately 3dB to 6dB.
  • Robustness: The results were consistent across various IRF values and photon counts, highlighting the network's robustness.
  • Computational Benefits: The asynchronous processing of spiking neurons led to faster throughput and reduced computational overhead.

Qualitative results also showed Spike-SPI achieving higher spatial acuity, resolving finer details in the reconstructed silhouettes compared to the ANN.

Limitations and Future Work

  • Accuracy Metrics Nuance: The paper notes that while Spike-SPI excels in spatial accuracy (IoU), some metrics that average over the entire image might favor the ANN. This is attributed to Spike-SPI's high precision but occasional large errors on a few pixels, which can skew average metrics.
  • Future Work: The authors suggest that future research could explore methods to account for dynamic scenes, especially given the new ability of asynchronous processing at lower photon return counts, which could help differentiate between static and dynamic elements. They also mention the potential for further optimization of spiking neuron parameters.

Summary updated

Abstract

A new approach for imaging that is solely based on the time of flight of photons coming from the entire imaged scene, combined with a novel machine learning algorithm for image reconstruction: a spiking convolutional neural network (SCNN) named Spike-SPI (Spiking - Single Pixel Imager). The approach uses a single point detector and the corresponding time-counting electronics, which provide the arrival time of photons in the form of spikes distributed over time. This data is transformed into a temporal histogram containing the number of photons per arrival time. A SCNN that converts the 1D temporal histograms into a 3D image (2D image with depth map) by exploiting the feature extraction capabilities of convolutional neural networks (CNNs), the high dimensional compressed latent space representations of a variational encoder-decoder network structure, and the asynchronous processing capabilities of a spiking neural network (SNN). The performance of the proposed SCNN is analysed to demonstrate the state-of-the-art feature extraction capabilities of CNNs and the low latency asynchronous processing of SNNs that offer both higher throughput and higher accuracy in image reconstruction from the ToF data, when compared to standard ANNs. The results of Spike-SPI show an increase in spatial accuracy of 15% over then ANN, using the Intersection of Union (IoU) for the objects in the scene. While also delivering a 100% increase over then ANN in object reconstruction signal to noise ratio (RSNR) from ~3dB to ~6dB. These results are also consistent across a range of IRF (Instrument Response Functions) values and photo counts, highlighting the robust nature of the new network structure. Moreover, the asynchronous processing nature of the spiking neurons allow for a faster throughput and less computational overhead, benefiting from the operational sparsity in the single point sensor.
BibTeX
@misc{kirkland2020imaging,
  author = {Paul Kirkland and Valentin Kapitány and Ashley Lyons and John J. Soraghan and Alex Turpin and Daniele Faccio and Gaetano Di Caterina},
  title = {{Imaging from temporal data via spiking convolutional neural networks}},
  year = {2020},
  doi = {10.1117/12.2573484},
}
Figure