Abstract
Calcium imaging is a powerful method to record the activity of neural populations in many species, but inferring spike times from calcium signals is a challenging problem. We compared multiple approaches using multiple datasets with ground truth electrophysiology and found that simple non-negative deconvolution (NND) outperformed all other algorithms on out-of-sample test data. We introduce a novel benchmark applicable to recordings without electrophysiological ground truth, based on the correlation of responses to two stimulus repeats, and used this to show that unconstrained NND also outperformed the other algorithms when run on "zoomed out" datasets of ∼10,000 cell recordings from the visual cortex of mice of either sex. Finally, we show that NND-based methods match the performance of a supervised method based on convolutional neural networks while avoiding some of the biases of such methods, and at much faster running times. We therefore recommend that spikes be inferred from calcium traces using simple NND because of its simplicity, efficiency, and accuracy.
Introduction
Two-photon calcium imaging has become a cornerstone of modern neuroscience, enabling researchers to monitor the activity of thousands of neurons simultaneously. However, the signal it produces—fluorescence—is an indirect and "blurry" proxy for neural activity. While an action potential (spike) occurs in a millisecond, the resulting calcium fluorescence rises and decays over hundreds of milliseconds. To understand neural computation, researchers must "deconvolve" these slow, noisy traces back into discrete spike times.
Recently, supervised learning approaches, specifically Convolutional Neural Networks (CNNs), have been claimed to outperform traditional methods. This paper investigates whether those gains are robust or simply the result of models overfitting to specific datasets. Furthermore, it examines whether complex "black box" models introduce artificial patterns that could lead to erroneous scientific conclusions.
Methods and Datasets
The study compared several flavors of Non-Negative Deconvolution (NND) against state-of-the-art supervised models:
- Unconstrained NND: A simple model assuming fluorescence is a convolution of a spike train and a calcium kernel (usually a simple exponential decay).
- L0/L1 Regularized NND: Versions that penalize spike counts or magnitudes to enforce sparsity.
- Supervised CNNs: Specifically the "Elephant" model, which was a top performer in public competitions.
The evaluation utilized a diverse range of data, including the GENIE and SPIKEFINDER datasets, which provide simultaneous "ground truth" electrophysiology. For large-scale recordings (e.g., ~10,000 neurons from mouse V1) lacking electrophysiology, the authors introduced a novel benchmark ($\rho_{stim}$) based on stimulus reliability: accurate deconvolution should yield similar spike estimates across repeated stimulus presentations.
Key Results
The analysis revealed a significant "generalization gap" for supervised models. While they perform well on data similar to their training sets, they often falter on "out-of-sample" data. Because experimental conditions—microscopes, indicators, and brain regions—vary, supervised models frequently fail in real-world settings.
In contrast, simple NND matched or outperformed supervised CNNs on nearly all out-of-sample benchmarks. NND proved surprisingly resilient to parameter mismatches; for instance, using a simple exponential decay kernel with an incorrect decay time had minimal impact on performance. Moreover, NND is roughly 100x faster than CNNs, processing thousands of cells in seconds on a standard CPU, whereas CNNs require expensive GPU resources.
Most alarmingly, supervised models can "hallucinate" temporal structures from their training data. If a CNN is trained on neurons with specific rhythmic patterns, it may impose that same rhythm onto new datasets where it does not exist, potentially leading to false scientific discoveries.
Significance and Conclusion
The experimental method that currently allows for recordings of the largest numbers of cells simultaneously is two-photon calcium imaging. However, use of this powerful method requires that neuronal firing times be inferred correctly from the large resulting datasets. Previous studies have claimed that complex supervised learning algorithms outperform simple deconvolution methods at this task. Unfortunately, these studies suffered from several problems and biases.
When we repeated the analysis, using the same data and correcting these problems, we found that simpler spike inference methods perform better. Even more importantly, we found that supervised learning methods can introduce artifactual structure into spike trains, which can in turn lead to erroneous scientific conclusions. Of the algorithms we evaluated, we found that an extremely simple method (NND) performed best in all circumstances tested, was much faster to run, and was insensitive to parameter choices. We therefore recommend that spikes be inferred from calcium traces using simple NND because of its simplicity, efficiency, and accuracy.
Summary updated