WormID-Bench: A Benchmark for Whole-Brain Activity Extraction in C. elegans

Jason Ken Adhinarta, Jinqian Dong, Tian-Xiao He, Daniel Sprague, Jia Wan, Hyun Jee Lee, Zikai Yu, Hang Lu, Eviatar Yemini, Saul Kato, Erdem Varol, Donglai WeibioRxiv 2025 · DOI

TL;DR

This paper introduces WormID-Bench, a comprehensive benchmark for extracting neural dynamics from whole-brain imaging of C. elegans. It addresses the critical challenges of neuron detection, identification, and tracking, revealing significant room for improvement in current computational methods and providing a standardized framework to drive future advancements.

What Problem Does This Paper Address?

Understanding how neural circuits function requires observing the activity of individual neurons within a complete nervous system. C. elegans is an ideal model organism for this due to its small, fully mapped nervous system and genetically identifiable neurons. Recent advances in 3D light microscopy allow us to capture whole-brain activity at the single-neuron level. However, turning these raw images into usable data for scientific analysis is a major bottleneck. This involves three key steps: accurately finding each neuron (detection), figuring out which specific neuron it is (identification), and following its activity over time (tracking). Current methods struggle with the variability across different imaging setups, biological differences between worms, and the dynamic nature of neural activity itself. This paper aims to standardize the evaluation of these steps and encourage the development of more robust and efficient computational tools.

Methods

WormID-Bench tackles the problem by breaking it down into three distinct tasks:

  • Neuron Detection: This involves identifying the location of individual neurons within 3D microscopy volumes. The paper explores two paradigms: directly inferring cell centroids from images or first segmenting the cells and then extracting centroids. They compare generalist pre-trained models like CellPose and microSAM against more specialized approaches.
  • Neuron Identification: Once neurons are detected, this task aims to assign each neuron to its specific anatomical class. This is facilitated by the NeuroPAL technique, which colors each neuron with a unique fluorescent RGB barcode. The problem is framed as either a point cloud registration task (aligning detected neurons to a template atlas) or a classification task (labeling each neuron into one of 302 classes). They evaluate established methods like Statistical Atlas, CRF ID, and fDNC.
  • Neuron Tracking: This task focuses on following the activity of identified neurons across time-lapse calcium imaging videos. The paper primarily investigates a "detect-to-track" approach, where neurons are detected in each frame and then linked across frames. They compare methods like Ultrack and 3DeeCellTracker.

Datasets

The core of WormID-Bench is a curated dataset comprising 118 C. elegans worms from five different laboratories. This dataset includes a mixture of NeuroPAL structural images and whole-brain calcium activity time series. Some of these datasets are newly introduced as part of the wormID corpus, while others are from previous publications. The data is standardized using the Neurodata Without Borders (NWB) format and is publicly available. The paper also details exclusion criteria for datasets with significant deformities or artifacts, resulting in 104 worms for detection/identification tasks and 36 for tracking. The evaluation uses a 5-fold cross-validation strategy to ensure robustness and generalizability.

Key Insights and Ideas

  • The "Annotation Bottleneck" is Real: The paper strongly emphasizes that while imaging hardware has advanced rapidly, the manual annotation of neural data remains a significant bottleneck, severely limiting the scale of scientific inquiry.
  • Cross-Laboratory Generalization is Crucial: Many existing methods perform well on data from a single lab but struggle when applied to data from different sources due to variations in imaging conditions, protocols, and biological variability. WormID-Bench's multi-laboratory dataset highlights this critical need for generalizable tools.
  • NeuroPAL is a Game-Changer (but not a Silver Bullet): The NeuroPAL technique, with its unique neuron coloring, greatly simplifies identification. However, it doesn't fully automate the process, as similar colors can appear in close proximity, and manual verification is still time-consuming.
  • Current State-of-the-Art Needs Improvement: The benchmark results consistently show that even the best current methods have substantial room for improvement across all three tasks, particularly in terms of sensitivity, specificity, and robustness to diverse conditions.

Key Results

  • Neuron Detection: Pre-trained models like CellPose showed reasonable out-of-the-box performance, but fine-tuning for 3D volumes was challenging. Micro-SAM also provided decent results, but both methods struggled with non-cellular structures and low-contrast neurons, leading to false positives and negatives.
  • Neuron Identification: Deep learning methods (CRF ID, fDNC) generally outperformed traditional methods (Statistical Atlas, CPD) when not retrained. However, retraining on the full WormID-Bench dataset significantly improved the performance of all methods, especially CRF ID, demonstrating the value of diverse training data. Accuracy was higher on datasets with fewer labels, indicating that identifying the most ambiguous neurons is the hardest part.
  • Neuron Tracking: Ultrack consistently outperformed 3DeeCellTracker in both detection and tracking accuracy. The paper suggests that 3DeeCellTracker's performance could be improved by using its "ensemble" mode and addressing issues with inconsistent neuron detection across frames.

Limitations and Future Work

The authors acknowledge that their curated dataset, while large and diverse, may not encompass all possible experimental conditions and optical hardware used by the C. elegans community. They envision future versions of the benchmark incorporating data from new labs and experimental setups. They also highlight the need for more advanced, domain-specific methods or foundation models trained on large corpora that can be fine-tuned for this specific organism. The research is focused on C. elegans, a phylogenetically distant organism from humans, minimizing concerns about direct negative societal impacts. The primary goal is to advance basic scientific understanding in invertebrate neuroscience.

Summary updated

Abstract

Abstract The nematode C. elegans is a premier model organism for studying neural circuit function due to its fully mapped connectome and genetically identifiable neurons. Recent advances in 3D light microscopy and fluorescent protein tagging have enabled whole-brain imaging at single-neuron resolution. However, extracting meaningful neural dynamics from these high-resolution recordings requires addressing three fundamental challenges: (i) accurate detection of individual neurons in fluorescence images, (ii) precise identification of neuron classes based on anatomical and colorimetric cues, and (iii) robust tracking of neurons over time in calcium imaging videos. To systematically evaluate these challenges, we introduce WormID-Bench, a large-scale, multi-laboratory dataset comprising 118 worms from five distinct research groups, along with standardized evaluation metrics for detection, identification, and tracking. Our benchmark reveals that existing computational approaches show substantial room for improvement in sensitivity, specificity, and generalization across diverse experimental conditions. By providing an open and reproducible benchmarking framework 1 , WormID-Bench aims to accelerate the development of high-throughput and scalable computational tools for whole-brain neural dynamics extraction in C. elegans , setting the stage for broader advancements in functional connectomics.
BibTeX
@misc{adhinarta2025wormid,
  author = {Jason Ken Adhinarta and Jinqian Dong and Tian-Xiao He and Daniel Sprague and Jia Wan and Hyun Jee Lee and Zikai Yu and Hang Lu and Eviatar Yemini and Saul Kato and Erdem Varol and Donglai Wei and Erdem Varol and Donglai Wei},
  title = {{WormID-Bench: A Benchmark for Whole-Brain Activity Extraction in <i>C. elegans</i>}},
  year = {2025},
  doi = {10.1101/2025.01.06.631621},
  howpublished = {bioRxiv},
}
Figure