Community-based benchmarking improves spike rate inference from two-photon calcium imaging data

Philipp Berens, Jeremy Freeman, Thomas Deneux, Nicolay Chenkov, Thomas McColgan, Artur Speiser, Jakob H. Macke, Srinivas C. Turaga, Patrick Mineault, Peter Rupprecht, Stephan Gerhard, Rainer Friedrich, Johannes Friedrich, Liam Paninski, Marius Pachitariu, Kenneth D. Harris, Ben Bolte, Timothy A. Machado, Dario L. Ringach, Jasmine T. Stone, Luke E. Rogerson, Nicolas J. Sofroniew, Jacob Reimer, Emmanouil Froudarakis, Thomas Euler, Miroslav Roman-Roson, Lucas Theis, Andreas S. Tolias, Matthias BethgebioRxiv 2017 · DOI

Abstract

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations. However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators. Different algorithms for estimating spike trains from noisy calcium measurements have been proposed in the past, but it is an open question how far performance can be improved. Here, we report the results of the spikefinder challenge, launched to catalyze the development of new spike inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.

Introduction

Two-photon calcium imaging is a cornerstone of modern systems neuroscience, enabling the simultaneous observation of thousands of neurons in vivo. However, the technique provides only an indirect measure of neural activity. Researchers observe fluorescent sensors that respond to calcium influx, which serves as a proxy for electrical action potentials (spikes). This indirect measurement introduces two primary challenges: temporal blurring, as calcium transients last significantly longer than the millisecond-scale spikes, and optical/biological noise. To reconstruct the underlying spike trains, researchers must "deconvolve" these slow, noisy traces. While numerous algorithms have been proposed, the field has lacked a standardized benchmark to evaluate their relative performance and determine the theoretical limits of spike inference.

Methods

The spikefinder challenge was organized to address this gap through community-driven benchmarking. The competition utilized five datasets comprising 92 recordings from 73 neurons in the mouse primary visual cortex and retina, all featuring simultaneous electrophysiological ground truth. These datasets included both synthetic dyes (OGB-1) and genetically encoded indicators (GCaMP6s).

Thirty-seven teams submitted algorithms, and the top ten were analyzed in depth. These methods generally fell into three categories:

  • Generative/Biophysical Models: Such as MLspike (Team 1), which explicitly models the physical process of calcium dynamics to invert the signal.
  • Supervised Deep Learning: Teams 2, 3, 4, 5, and 8 utilized Deep Neural Networks (DNNs), including CNNs, LSTMs, and ResNets, to learn the mapping from fluorescence to spikes.
  • Hybrid Approaches: Team 9 combined these paradigms by training a neural network on large-scale simulated data generated from biophysical models.

To ensure rigorous evaluation, a portion of the data was kept private for final testing, preventing algorithms from simply overfitting to the training sets.

Results

The challenge successfully drove significant improvements in spike inference. The top-performing algorithms achieved an average correlation with ground truth of 0.44, a substantial increase from the previous state-of-the-art (STM) baseline of 0.36. This represents a >40% improvement in variance explained ($r^2$).

Performance analysis of the spikefinder challenge. (A) Correlation coefficients across datasets for the top 10 teams. (B) Improvement over the STM baseline for OGB-1 and GCaMP6s indicators. (C) P-values for pairwise comparisons between teams. (D) Generalization performance comparing training vs. test sets.

A key finding was the convergence of different algorithmic approaches. Despite their diverse mathematical foundations, the top algorithms produced highly correlated predictions, suggesting that the field may be approaching a performance ceiling dictated by the signal-to-noise ratio and physics of current calcium indicators. While DNN-based methods were highly effective, they showed a greater tendency toward overfitting compared to simpler models (Panel D). Notably, Team 10 demonstrated that a very simple linear-nonlinear model could achieve competitive performance, proving that complex architectures are not always a prerequisite for success.

Discussion

The spikefinder challenge demonstrates that standardized benchmarks can catalyze algorithmic development. While performance has improved, several challenges remain. Most algorithms still require dataset-specific tuning, highlighting the need for better "transfer learning" capabilities to make these tools truly "plug-and-play" across different laboratories. Additionally, the theoretical "Bayes-optimal" limit for spike inference given current imaging technology remains unknown. Future benchmarks should focus on these generalization hurdles and the standardization of preprocessing pipelines to further refine the accuracy of neural activity reconstruction.

Summary updated

Abstract

In recent years, two-photon calcium imaging has become a standard tool to probe the function of neural circuits and to study computations in neuronal populations. However, the acquired signal is only an indirect measurement of neural activity due to the comparatively slow dynamics of fluorescent calcium indicators. Different algorithms for estimating spike trains from noisy calcium measurements have been proposed in the past, but it is an open question how far performance can be improved. Here, we report the results of the spikefinder challenge, launched to catalyze the development of new spike inference algorithms through crowd-sourcing. We present ten of the submitted algorithms which show improved performance compared to previously evaluated methods. Interestingly, the top-performing algorithms are based on a wide range of principles from deep neural networks to generative models, yet provide highly correlated estimates of the neural activity. The competition shows that benchmark challenges can drive algorithmic developments in neuroscience.
BibTeX
@misc{berens2017community,
  author = {Philipp Berens and Jeremy Freeman and Thomas Deneux and Nicolay Chenkov and Thomas McColgan and Artur Speiser and Jakob H. Macke and Srinivas C. Turaga and Patrick Mineault and Peter Rupprecht and Stephan Gerhard and Rainer Friedrich and Johannes Friedrich and Liam Paninski and Marius Pachitariu and Kenneth D. Harris and Ben Bolte and Timothy A. Machado and Dario L. Ringach and Jasmine T. Stone and Luke E. Rogerson and Nicolas J. Sofroniew and Jacob Reimer and Emmanouil Froudarakis and Thomas Euler and Miroslav Roman-Roson and Lucas Theis and Andreas S. Tolias and Matthias Bethge},
  title = {{Community-based benchmarking improves spike rate inference from two-photon calcium imaging data}},
  year = {2017},
  doi = {10.1101/177956},
  howpublished = {bioRxiv},
}
Figure