Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations

Nancy X. R. Wang, Jared D. Olson, Jeffrey G. Ojemann, Rajesh P. N. Rao, Bingni W. BruntonFrontiers in Human Neuroscience 2016 · DOI

Abstract

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.

Introduction: The Challenge of Real-World BCIs

Most current Brain-Computer Interfaces (BCIs) are "lab-bound," relying on supervised learning paradigms where subjects perform highly repetitive, cued tasks. While effective in controlled settings, this approach creates a significant bottleneck for clinical applications. Manual labeling of real-world, "naturalistic" behavior is exhausting, prone to human error, and fails to scale to the weeks of continuous data required for truly useful assistive technologies.

Furthermore, neural signals in unconstrained environments are inherently messy. A subject may be eating, sleeping, or interacting with others. To transition BCIs from the laboratory to the living room, systems must learn to recognize these states automatically, without constant human intervention.

Methods: An Unsupervised Multi-Modal Pipeline

The researchers developed a multi-modal unsupervised framework that integrates intracranial brain recordings (Electrocorticography, or ECoG) with automated environmental sensors.

Data Collection and Feature Extraction

The study utilized a unique dataset from six clinical patients undergoing presurgical epilepsy monitoring. Continuous ECoG data (approximately 100 electrodes per subject) were recorded over at least one week, synchronized with 24/7 video and audio from the hospital room.

Multi-modal recording setup showing synchronized audio, video (day/night), and electrode data.

Raw neural signals were transformed into power spectral features (1–52 Hz). To manage the high dimensionality of the data, Principal Component Analysis (PCA) was employed to reduce the features to the top 50 components.

Hierarchical Clustering and Automated Annotation

Instead of predefined behavioral categories, the team used hierarchical k-means clustering to identify coherent patterns within the neural data. This produced a tree of clusters representing distinct brain states. To interpret these clusters, the system employed:

  • Computer Vision: Speeded Up Robust Features (SURF) to detect movement levels in video frames.
  • Speech Processing: Audio power analysis focused on the human speech frequency range (100–3500 Hz).

The system automatically labeled neural clusters by correlating their temporal activation with these automated motion and speech metrics. For instance, a cluster consistently active during periods of high video motion was labeled "Movement."

Key Results: Stability and Spatial Mapping

The unsupervised approach proved remarkably robust. Automated video and audio labeling achieved 74% accuracy for movement and 75% for speech when validated against a "gold standard" of manual human labels.

Neural Decoding and Generalization

The system successfully identified three primary behavioral states: Moving, Speaking, and Resting. A critical finding was the stability of these decoders; a model trained on Day 1 could accurately predict behavior on Day 4 without additional video or audio input. This suggests that the neural signatures for these fundamental behaviors remain stable over several days in naturalistic settings.

Automated Functional Mapping

By projecting the discovered clusters back onto the cortical surface, the researchers found that the "Speech" clusters localized to known auditory and language areas, while "Movement" clusters aligned with motor areas.

Automated functional mapping: neural clusters projected onto the cortical surface.

This capability suggests that we can potentially map a patient's brain functions simply by observing their natural daily activities, bypassing the need for rigorous experimental tasks.

Discussion: Overcoming Environmental Noise

While the results are promising, the "messiness" of naturalistic data presents ongoing challenges. The current movement detection system occasionally struggles to distinguish between the patient's movements and those of hospital staff (the "Nurse Problem"). Future iterations may utilize depth cameras (e.g., Kinect) for better subject segmentation. Similarly, audio specificity could be improved with directional microphones or speaker identification to distinguish the patient's voice from background noise or television.

Despite these limitations, this work represents a major step toward "passive BCIs" that can adapt to a user's life in real-time, providing a scalable path for robust assistive technologies in non-experimental environments.

Summary updated

Abstract

Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.
BibTeX
@inproceedings{wang2016unsupervised,
  author = {Nancy X. R. Wang and Jared D. Olson and Jeffrey G. Ojemann and Rajesh P. N. Rao and Bingni W. Brunton},
  title = {{Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2016},
  doi = {10.3389/fnhum.2016.00165},
}
Figure