Robustness of spike deconvolution for calcium imaging of neural spiking

Marius Pachitariu, Carsen Stringer, Kenneth D. HarrisbioRxiv 2017 · DOI

TL;DR

This paper argues that simple, unconstrained Non-Negative Deconvolution (NND) is the most robust and efficient method for inferring neural spiking activity from calcium imaging data. The authors introduce a new benchmark that doesn't require ground truth electrophysiology, demonstrating NND's superiority even on large-scale datasets and its comparable performance to complex deep learning models, but with significantly faster execution times.

What Problem Does This Paper Address?

Calcium imaging is a widely used technique to monitor the activity of large populations of neurons. However, the fluorescence signal measured is an indirect and noisy proxy for actual neuronal firing (spikes). Accurately inferring these spike times from the fluorescence traces – a process called "spike deconvolution" – is crucial for understanding neural computations. While various sophisticated algorithms exist, including supervised machine learning approaches, their performance and generalizability, especially to new datasets or experimental conditions, remain a challenge. This paper aims to systematically compare different deconvolution methods, identify the most robust and accurate approach, and propose a more practical benchmarking strategy.

Methods

The core of the paper involves comparing several spike deconvolution algorithms. The primary method is Non-Negative Deconvolution (NND), a relatively simple approach that assumes the fluorescence signal is a convolution of the underlying spike train with a calcium indicator's response kernel. The authors specifically focus on an unconstrained version of NND (where no penalty is applied to the sparsity of the inferred spike train) and compare it to NND with L0 and L1 sparsity penalties.

They also evaluate state-of-the-art supervised methods, including convolutional neural networks (CNNs) like the "Elephant" model, which were winners of the "spikefinder" challenge.

A key methodological contribution is the development of a novel benchmark that does not require electrophysiological ground truth. This benchmark relies on the correlation of a neuron's responses to two repeated stimulus presentations. The idea is that if the deconvolution successfully captures the underlying spiking activity, the reconstructed spike trains from the two repeats should be highly correlated. This allows for benchmarking on datasets where ground truth electrophysiology is unavailable, which is common in many calcium imaging experiments.

Datasets

The study utilizes two main types of datasets:

  • Datasets with Ground Truth Electrophysiology:
    • GENIE: A collection of five datasets from the GENIE project, used in the original descriptions of GCaMP calcium sensors.
  • SPIKEFINDER: Five datasets from the "spikefinder" challenge, analyzed by Theis et al. (2016).
  • New Benchmark Datasets (without Ground Truth):
    • The authors use their own datasets from approximately 10,000 neurons in awake mice, recorded during visual cortex imaging experiments. These datasets are used to validate the new "repeat similarity" benchmark.

The new benchmark is designed to assess deconvolution performance by measuring the Spearman correlation ($\sigma_{stim}$) of deconvolved responses to two repeats of an ensemble of stimuli. This metric approximates the proportion of signal-related variance in the trial-averaged responses, providing a measure of how well the deconvolution reconstructs the true spiking signal.

Key Insights and Ideas

  • Simplicity Wins: Unconstrained Non-Negative Deconvolution (NND) consistently outperforms more complex penalized versions of NND and even state-of-the-art supervised methods when evaluated appropriately.
  • Robustness to Assumptions: NND is remarkably robust to variations in the assumed shape and timescale of the calcium response kernel. Using a simple exponential decay kernel is sufficient and often better than using more biologically complex kernels or kernels estimated directly from ground truth data.
  • Generalization Matters: Supervised methods, while performing well on "in-sample" data (data similar to their training set), can struggle to generalize to new datasets or experimental conditions. This is a significant limitation for real-world applications.
  • New Benchmark Utility: The proposed "repeat similarity" benchmark is a valuable tool for evaluating deconvolution methods without the need for simultaneous electrophysiology, making it broadly applicable.
  • CNNs Overfit: Convolutional Neural Networks (CNNs), while powerful, can overfit to the specific statistics of the training data, leading to potentially misleading conclusions about neural dynamics when applied to new datasets.
  • Speed Advantage: NND is orders of magnitude faster than CNNs, making it a more practical choice for analyzing large calcium imaging datasets.

Key Results

  • NND Outperforms: On both GENIE and SPIKEFINDER datasets, unconstrained NND, after simple post-processing (timelag adjustment and smoothing), achieved higher correlation with ground truth spike trains than supervised methods like "stm" and "oopsi".
  • Parameter Insensitivity: Varying the kernel timescale by a factor of 2 had little effect on NND's performance. Adding a rising time component to the kernel did not improve results.
  • New Benchmark Validation: On the new "repeat similarity" benchmark, unconstrained NND again outperformed supervised methods, including the CNN approach.
  • CNN vs. NND Generalization: When tested on out-of-sample data, the performance of supervised CNNs and unconstrained NND was nearly identical.
  • Speed: NND runs orders of magnitude faster than CNNs, making it significantly more practical for large-scale analyses. For example, CNNs take minutes per dataset on a high-end GPU, while NND runs efficiently on standard CPUs.

Limitations and Future Work

  • Kernel Estimation: While NND is robust to kernel parameters, the paper notes that automated estimation of kernel parameters can be counterproductive. The authors recommend using fixed, literature-derived timescales. Future work could explore more robust kernel estimation strategies.
  • New Methods: The authors acknowledge that it remains to be seen if future methods can significantly outperform simple NND, but suggest that marginal improvements might be overshadowed by NND's simplicity and interpretability.
  • OASIS Implementation: The paper highlights OASIS as an efficient algorithm for NND. The authors also mention that their Suite2p pipeline includes wrappers for OASIS and L0-based deconvolution, suggesting ongoing development and integration of these tools.
  • Further Benchmark Applications: The authors encourage the use of their new benchmark for evaluating deconvolution methods on users' own data.

Summary updated

Abstract

Abstract Calcium imaging is a powerful method to record the activity of neural populations, but inferring spike times from calcium signals is a challenging problem. We compared multiple approaches using multiple datasets with ground truth electrophysiology, and found that simple non-negative deconvolution (NND) outperformed all other algorithms. We introduce a novel benchmark applicable to recordings without electrophysiological ground truth, based on the correlation of responses to two stimulus repeats, and used this to show that unconstrained NND also outperformed the other algorithms when run on “zoomed out” datasets of ~10,000 cell recordings. Finally, we show that NND-based methods match the performance of a supervised method based on convolutional neural networks, while avoiding some of the biases of such methods, and at much faster running times. We therefore recommend that spikes be inferred from calcium traces using simple NND, due to its simplicity, efficiency and accuracy.
BibTeX
@misc{pachitariu2017robustness,
  author = {Marius Pachitariu and Carsen Stringer and Kenneth D. Harris},
  title = {{Robustness of spike deconvolution for calcium imaging of neural spiking}},
  year = {2017},
  doi = {10.1101/156786},
  howpublished = {bioRxiv},
}

Loading related content...

Figure