AJILE12: Long-term naturalistic human intracranial neural recordings and pose

Steven Peterson, Satpreet H. Singh, Ben Dichter, Michael R. Scheid, Rajesh P. N. Rao, Bingni W. BruntonbioRxiv 2021 · DOI

TL;DR

This paper introduces AJILE12, the largest publicly available dataset of synchronized intracranial neural recordings (ECoG) and upper body pose trajectories from 12 human participants during naturalistic daily activities. This rich dataset, spanning 55 days, aims to bridge the gap between highly controlled lab experiments and real-world human movement neuroscience, enabling more robust brain-computer interfaces and a deeper understanding of motor control.

What Problem Does This Paper Address?

Understanding how the brain controls movement is a fundamental goal in neuroscience. However, much of our current knowledge comes from studies involving highly constrained, repetitive tasks like reaching for a target in a lab. While these studies have been invaluable, it's unclear how well the findings generalize to the fluid, complex, and often unpredictable movements we perform in everyday life. Capturing this naturalistic movement data, especially with high-resolution neural recordings, has been challenging due to the difficulty and labor-intensiveness of tracking human pose and synchronizing it with neural signals over extended periods. This gap limits our ability to develop advanced neuroprosthetics and truly understand the neural basis of natural human behavior.

Methods

The researchers collected data from 12 participants undergoing clinical epilepsy monitoring. They utilized electrocorticography (ECoG) to record neural activity at 500 Hz from at least 64 electrodes per participant. Crucially, they employed markerless pose estimation using DeepLabCut, a computer vision technique, to automatically track 9 upper-body keypoints (including wrists, elbows, and shoulders) from video footage. This automated approach allowed them to process an enormous number of video frames (118 million) and synchronize neural data with pose trajectories.

To ensure data quality and facilitate analysis, they implemented several processing steps:

  • ECoG Processing: DC drift removal, artifact rejection, band-pass filtering (1-200 Hz), notch filtering, downsampling to 500 Hz, and common median re-referencing.
  • Pose Estimation: Training participant-specific DeepLabCut models on manually annotated frames and applying them to all video frames.
  • Synchronization: Aligning ECoG and pose data using video timestamps.
  • Movement Event Identification: Using a hidden semi-Markov model on wrist trajectories to identify discrete movement initiation events, followed by rigorous filtering based on duration, confidence scores, and trajectory shape.
  • Metadata Generation: Extracting numerous quantitative features for each movement event (e.g., reach magnitude, duration, speed), annotating coarse behavioral states (e.g., sleep/rest, active, inactive) from video, and including electrode-level features.

Datasets

The primary contribution of this paper is the introduction of the AJILE12 dataset. This dataset is described as the largest publicly available human neurobehavioral dataset, featuring synchronized intracranial neural recordings (ECoG) and upper body pose trajectories. It comprises 55 days of semi-continuous recordings from 12 participants, totaling approximately 1280 hours of ECoG data and 118 million video frames for pose estimation. The data is made available in the Neurodata Without Borders (NWB) format on The DANDI Archive, a public data repository.

Key Insights and Ideas

  • Bridging the Lab-to-Real-World Gap: The AJILE12 dataset represents a significant step towards studying human movement in more ecologically valid settings, moving beyond the limitations of highly controlled laboratory paradigms.
  • The Power of Automated Pose Estimation: The use of DeepLabCut was critical in enabling the collection of such a large-scale dataset by automating the laborious process of pose annotation, which has historically been a bottleneck.
  • Rich Metadata for Deeper Analysis: The inclusion of detailed metadata, including quantified movement events, behavioral states, and electrode features, allows for multi-faceted analyses that go beyond simple neural decoding.
  • Standardized Data Format for Reusability: Releasing the data in the NWB format promotes findability, accessibility, interoperability, and reusability (FAIR principles), making it easier for other researchers to utilize and build upon this resource.
  • Browser-Based Exploration: The development of an interactive dashboard significantly lowers the barrier to entry for exploring the dataset, allowing researchers to visualize data without needing to download massive files.

Key Results

The paper details the successful collection and processing of the AJILE12 dataset, highlighting its scale and comprehensiveness:

  • Data Volume: 55 days of recordings, ~1280 hours of ECoG data, and 118 million video frames processed for pose estimation.
  • Neural Recording Quality: ECoG data sampled at 500 Hz from 64-126 electrodes per participant. Technical validation showed consistent spectral power across participants and time windows, indicating good signal quality.
  • Pose Estimation Accuracy: Markerless pose estimation models achieved root-mean-square errors of 1.52 ± 0.12 pixels on training sets and 5.71 ± 1.90 pixels on holdout sets, which is considered within an acceptable tolerance for this type of data.
  • Movement Events: Thousands of verified wrist movement initiation events were identified and cataloged.
  • Data Accessibility: The dataset is publicly available on The DANDI Archive, and a browser-based dashboard is provided for exploration.

Limitations and Future Work

  • Opportunistic Data Collection: The data was collected opportunistically during clinical epilepsy monitoring, meaning participants were confined to a hospital bed and their activities were limited to everyday tasks within that environment. This is more naturalistic than a lab setting but not fully representative of all real-world movements.
  • Pose Estimation Limitations: While DeepLabCut is powerful, the paper acknowledges that manual annotations are susceptible to human error, and the pose estimation errors, though acceptable, are still present. Occlusions can also affect pose estimation accuracy.
  • Behavioral Annotation Granularity: Coarse behavioral labels were annotated every ~3 minutes, which might miss finer-grained behavioral shifts.
  • Future Work: The authors suggest that the AJILE12 dataset can be used for a wide range of future analyses, including developing more robust brain-computer interfaces for real-world use, investigating neurobehavioral variability, and exploring complex motor behaviors with modern machine learning techniques. The dataset's size and richness are expected to enable large-scale explorations that were previously not possible.

Summary updated

Abstract

ABSTRACT Understanding the neural basis of human movement in naturalistic scenarios is critical for expanding neuroscience research beyond constrained laboratory paradigms. Here, we describe our Annotated Joints in Long-term Electrocorticography for 12 human participants (AJILE12) dataset, the largest human neurobehavioral dataset that is publicly available; the dataset was recorded opportunistically during passive clinical epilepsy monitoring. AJILE12 includes synchronized intracranial neural recordings and upper body pose trajectories across 55 semi-continuous days of naturalistic movements, along with relevant metadata, including thousands of wrist movement events and annotated behavioral states. Neural recordings are available at 500 Hz from at least 64 electrodes per participant, for a total of 1280 hours. Pose trajectories at 9 upper-body keypoints were estimated from 118 million video frames. To facilitate data exploration and reuse, we have shared AJILE12 on The DANDI Archive in the Neurodata Without Borders (NWB) data standard and developed a browser-based dashboard.
BibTeX
@misc{peterson2021ajile,
  author = {Steven Peterson and Satpreet H. Singh and Ben Dichter and Michael R. Scheid and Rajesh P. N. Rao and Bingni W. Brunton},
  title = {{AJILE12: Long-term naturalistic human intracranial neural recordings and pose}},
  year = {2021},
  doi = {10.1101/2021.07.26.453884},
  howpublished = {bioRxiv},
}
Figure