A detailed behavioral, videographic, and neural dataset on object recognition in mice

Chris C. Rodgers

Abstract

Mice adeptly use their whiskers to touch, recognize, and learn about objects in their environment. This ability is enabled by computations performed by populations of neurons in the somatosensory cortex. To understand these computations, we trained mice to use their whiskers to recognize different shapes while we recorded activity in the barrel cortex, which processes whisker input. Here, we present a large dataset of high-speed video of the whiskers, along with rigorous tracking of the entire extent of multiple whiskers and every contact they made on the shape. We used spike sorting to identify individual neurons, which responded with precise timing to whisker contacts and motion. These data will be useful for understanding the behavioral strategies mice use to identify objects by touch, as well as the neuronal responses that mediate those strategies. More generally, our carefully curated labeled data could be used to develop new computer vision algorithms for tracking body posture, or for extracting responses of individual neurons from large-scale neural recordings.

Introduction

In systems neuroscience, researchers often study sensory processing and motor control as separate silos. However, real-world perception is fundamentally "active": a mouse does not simply wait for a touch; it actively sweeps its whiskers to explore its environment. This paper introduces a massive, open-source dataset that synchronizes high-speed video of mouse whisking with high-density neural recordings in the somatosensory cortex. By providing over 88 hours of precisely tracked whisker movements and 33 hours of neural data, the authors offer a unique resource for studying the complex loop where motor actions drive sensory perception.

The primary challenge in capturing this "closed-loop" behavior is synchronizing three disparate data streams: high-speed video, behavioral logs, and high-density neural recordings. Historically, datasets combining all three with high precision have been rare, hindering our understanding of how specific whisker movements translate into neural computations and behavioral decisions.

Methods

The researchers trained mice on two tasks: shape discrimination (distinguishing concave from convex shapes) and shape detection. Key technical highlights include:

  • Iterative Pose Tracking: To track flexible, continuous whiskers, the team used a deep convolutional neural network. They employed an iterative "challenge frame" approach, identifying frames where the network struggled (e.g., during intense whisking or motion blur), manually correcting them, and retraining the model. This pushed the error rate below 0.2%.
  • High-Density Electrophysiology: 64-channel silicon probes were used to record from all layers of the barrel cortex simultaneously.
  • Hardware-Level Synchronization: To align the clocks of the video camera (200 fps), the neural logger (30 kHz), and the behavioral controller, a single synchronization pulse was used. This pulse simultaneously marked the neural data and turned off the whisker backlight for 133ms, creating an unambiguous "start" frame in the video.

Data Organization and Standardization

The entire dataset is released via the Dandi Archive (Dandiset 000231) in the Neurodata Without Borders (NWB) format. The scope includes 15 mice, 115 sessions, and over 18,000 trials. The use of the NWB format ensures that the data is "plug-and-play" for researchers using Python or MATLAB, lowering the barrier to entry for complex secondary analyses.

Overview of the NWB file structure and its constituent fields.

As shown in the figure above, the NWB structure organizes the data into logical groups such as acquisition for raw recordings, processing for behavioral events, and trials for session parameters.

Key Results and Insights

Analysis of this data has revealed several critical insights into cortical function:

  • Task-Dependent Coding: The barrel cortex changes its strategy based on the goal. In discrimination tasks, neurons compare contacts across different whiskers, whereas in detection tasks, they respond to the total number of contacts.
  • Orthogonal Subspaces: Sensory variables (touch) and motor variables (whisker movement) are represented in nearly orthogonal neural subspaces, allowing the brain to distinguish between self-generated movement and external contact.
  • Sub-frame Precision: The continuous alignment method achieved sub-frame synchronization, essential for mapping millisecond-scale neural spikes to specific whisker contacts.

Discussion and Future Directions

While the dataset is extensive, it has limitations, such as the trimming of whiskers to a single row (the C-row) to simplify tracking. This does not capture the full 3D complexity of a natural whisker array. Furthermore, the current neural data is limited to the barrel cortex; future studies could expand this paradigm to the thalamus or motor cortex to observe decision-making signal propagation.

Ultimately, this dataset serves as a "gold mine" for both neuroscientists and computer scientists. It provides a curated and labeled foundation for developing new pose-tracking algorithms or spike-sorting techniques, moving the field closer to a holistic understanding of active sensing.

Summary updated

Abstract

Abstract Mice adeptly use their whiskers to touch, recognize, and learn about objects in their environment. This ability is enabled by computations performed by populations of neurons in the somatosensory cortex. To understand these computations, we trained mice to use their whiskers to recognize different shapes while we recorded activity in the barrel cortex, which processes whisker input. Here, we present a large dataset of high-speed video of the whiskers, along with rigorous tracking of the entire extent of multiple whiskers and every contact they made on the shape. We used spike sorting to identify individual neurons, which responded with precise timing to whisker contacts and motion. These data will be useful for understanding the behavioral strategies mice use to identify objects by touch, as well as the neuronal responses that mediate those strategies. More generally, our carefully curated labeled data could be used to develop new computer vision algorithms for tracking body posture, or for extracting responses of individual neurons from large-scale neural recordings.
BibTeX
@inproceedings{rodgers2022detailed,
  author = {Chris C. Rodgers},
  title = {{A detailed behavioral, videographic, and neural dataset on object recognition in mice}},
  booktitle = {Scientific},
  year = {2022},
  doi = {10.1038/s41597-022-01728-1},
}
Figure