Abstract
Recent advances in calcium imaging enable simultaneous recordings of up to a million neurons in behaving animals, producing datasets of unprecedented scales. Although individual neurons and their activity traces can be extracted from these videos with automated algorithms, the results often require human curation to remove false positives, a laborious process called cell sorting. To address this challenge, we introduce ActSort, an active-learning algorithm for sorting large-scale datasets that integrates features engineered by domain experts together with data formats with minimal memory requirements. By strategically bringing outlier cell candidates near the decision boundary up for annotation, ActSort reduces human labor to about 1–3% of cell candidates and improves curation accuracy by mitigating annotator bias. To facilitate the algorithm’s widespread adoption among experimental neuroscientists, we created a user-friendly software and conducted a first-of-its-kind benchmarking study involving about 160,000 annotations. Our tests validated ActSort’s performance across different experimental conditions and datasets from multiple animals. Overall, ActSort addresses a crucial bottleneck in processing large-scale calcium videos of neural activity and thereby facilitates systems neuroscience experiments at previously inaccessible scales.
Introduction
In modern systems neuroscience, calcium imaging has revolutionized our ability to observe neural population dynamics, allowing researchers to record the activity of up to a million neurons simultaneously. However, the raw data produced by these experiments—high-speed fluorescence movies—requires significant processing to extract meaningful signals. While automated algorithms exist to identify neurons and their activity traces, they are prone to misidentifying motion artifacts, neuropil contamination, or noise as genuine neural activity.
To ensure data quality, researchers typically perform "cell sorting," a manual curation process where experts review thousands of candidates to separate real neurons from false positives. This process has become a massive bottleneck in the research pipeline; it is incredibly laborious, often requiring dozens of hours per dataset, and is susceptible to human error and "annotator bias" as fatigue sets in. ActSort was developed to automate this curation process, maintaining the nuanced judgment of a human expert while slashing the required manual labor by up to 99%.
Methodology
The ActSort framework is built upon a three-part pipeline designed for efficiency and accuracy:
- Memory-Efficient Preprocessing: To handle multi-terabyte datasets on standard hardware, ActSort avoids loading entire movies into RAM. Instead, it compresses data by extracting small "snapshots" of movie frames only during periods when a cell candidate is active. This allows the software to remain performant even on standard laptops.
- Expert-Engineered Features: The researchers designed 76 specific spatial, temporal, and spatiotemporal features that mimic the visual cues used by human experts. Interestingly, these hand-crafted features were found to outperform generic deep-learning models (such as ResNet-50) for this specific task, particularly in noisier one-photon (1p) imaging environments.
- Discriminative-Confidence Active Learning (DCAL): This core query algorithm determines which cell candidates require human intervention. DCAL balances two strategies: Confidence Active Learning (CAL), which selects samples where the classifier is most uncertain (near the decision boundary), and Discriminative Active Learning (DAL), which selects samples most different from the existing labeled set to ensure diversity. The system adaptively weights these strategies, prioritizing exploration (DAL) early on and refinement (CAL) as the model matures.
Experimental Setup and Benchmarking
To validate the tool, the authors conducted a massive benchmarking study involving approximately 160,000 manual annotations of 40,000 cell candidates. The dataset spanned five different mice and utilized both one-photon (1p) and two-photon (2p) mesoscopes. By having each cell labeled by multiple independent annotators, the team established a "majority vote" gold standard, which also allowed them to quantify the inherent disagreement between human experts.
Key Insights and Results
Human Inconsistency and Standardization
The study revealed that human experts often disagree on "edge case" cells, with an Intraclass Correlation Coefficient of approximately 0.56. ActSort helps standardize results across datasets by providing a consistent classification logic that mitigates individual human biases.
Labor Efficiency and Convergence
ActSort demonstrated massive labor savings, reaching human-level accuracy after a researcher labeled only 1–3% of the total cell candidates. In large-scale datasets, the model typically converged after roughly one hour of human effort. In contrast, traditional random sampling methods required an order of magnitude more work to achieve comparable accuracy.
Generalization and Biological Integrity
The system showed strong cross-animal generalization; a classifier pre-trained on one subject could be fine-tuned for a new subject with as little as 10–15 minutes of additional annotation. Furthermore, the paper highlights the biological importance of rigorous sorting: without it, noise artifacts like brain motion can inadvertently correlate with animal behavior (e.g., running speed), leading to fundamentally incorrect biological conclusions.
Limitations and Future Directions
While ActSort significantly optimizes the curation process, it remains a quality control tool for existing extraction pipelines. It cannot recover neurons that upstream algorithms (like EXTRACT or Suite2p) fail to detect initially. Currently, the tool focuses on binary classification (cell vs. non-cell), though future iterations may expand to multi-class sorting, such as distinguishing between somas and dendrites. Additionally, as the model's performance is tied to the quality of initial human labels, future research will explore how varying levels of human expertise impact the active learning trajectory.
By addressing the curation bottleneck, ActSort enables systems neuroscience experiments at scales that were previously inaccessible due to the sheer volume of manual labor required.
Summary updated