Abstract
Brain activation can reflect semantic information elicited by natural words and concepts. Increasing research has been conducted on decoding such neural activation patterns using representational semantic models. However, prior work decoding semantic meaning from neurophysiological responses has been largely limited to ECoG, fMRI, MEG, and EEG techniques, each having its own advantages and limitations. More recently, functional near-infrared spectroscopy (fNIRS) has emerged as an alternative hemodynamic-based approach and possesses a number of strengths. We investigate brain decoding tasks under the help of fNIRS and empirically compare fNIRS with fMRI. Primarily, we find that: 1) like fMRI scans, activation patterns recorded from fNIRS encode rich information for discriminating concepts, but show limits on the possibility of decoding fine-grained semantic clues; 2) fNIRS decoding shows robustness across different brain regions, semantic categories and even subjects; 3) fNIRS has higher accuracy being decoded based on multi-channel patterns as compared to single-channel ones. Our findings prove that fNIRS has the potential to promote a deep integration of NLP and cognitive neuroscience.
Introduction
Brain decoding—the process of mapping neural activity to linguistic meanings—has traditionally relied on high-fidelity but restrictive modalities. Functional Magnetic Resonance Imaging (fMRI) provides exceptional spatial resolution but requires subjects to remain immobile within a large, expensive scanner. Conversely, Electroencephalography (EEG) is portable but measures electrical activity, which is often noisy and lacks the hemodynamic context associated with complex semantic processing.
This research explores the utility of functional near-infrared spectroscopy (fNIRS) as a middle ground. By using near-infrared light to measure cortical blood oxygenation, fNIRS offers a portable, motion-tolerant, and cost-effective alternative to fMRI. The central question is whether the signal captured by fNIRS is sufficiently rich to distinguish between specific semantic concepts.
Methodology
The study utilizes a two-step experimental design: a pilot study for hyperparameter tuning and a full-scale experiment to evaluate decoding performance.
Experimental Setup
Participants were presented with visual and auditory stimuli representing 50 distinct concepts across 10 semantic categories (e.g., animals, tools, buildings). Neural responses were recorded using an fNIRS cap, focusing on oxygenated hemoglobin (HbO) levels.
Decoding Model
The researchers employed Ridge Regression to map HbO patterns to 50-dimensional GloVe word embeddings. Two decoding strategies were compared:
- Single-Channel Decoding (SCD): Analyzing individual sensors independently.
- Multi-Channel Decoding (MCD): Analyzing patterns across the entire sensor array to capture distributed neural representations.
Dataset
The authors released the largest fNIRS brain-decoding dataset to date, providing a benchmark for the community. Results were compared against the Mitchell et al. (2008) fMRI dataset to assess relative performance.
Results and Discussion
Multi-Channel Superiority and Temporal Dynamics
The study confirmed that Multi-Channel Decoding (MCD) significantly outperforms SCD, reflecting the distributed nature of semantic storage in the brain. Peak decoding accuracy was observed between 6.5 and 7.5 seconds post-stimulus, aligning with the biological hemodynamic response function.
Cross-Subject Generalizability
A key finding was the model's ability to generalize across individuals. As the number of training subjects increased, the Mean Squared Error (MSE) for fNIRS decoding decreased, suggesting a shared "neural vocabulary" for concepts across different people.
Comparison with fMRI
While fMRI remains superior for fine-grained distinctions (e.g., differentiating between two types of clothing) due to its depth penetration and higher spatial resolution, fNIRS proved remarkably robust. fNIRS signals were consistently decodable across various cortical regions (temporal, occipital, and parietal), whereas fMRI performance is more localized.
Conclusion
fNIRS represents a viable and flexible tool for brain-to-language decoding. While it faces limitations regarding penetration depth and sensitivity to physical factors like hair thickness, its portability and robustness make it a promising candidate for real-world neurotechnology applications. Future work will focus on real-time decoding and the integration of advanced deep learning architectures to further bridge the gap between hemodynamic signals and semantic understanding.
Summary updated