Visual Semantic Encoding and Identification of Naturalistic Movies via High-Density Diffuse Optical Tomography

Wiete Fehner, Morgan Fogarty, Jerry Tang, Dana Wilhelm, Aahana Bajracharya, Zachary E. Markow, Andrew Hines, Jason W. Trobaugh, Alexander G. Huth, Joseph P. Culver2025 · DOI

TL;DR

This paper demonstrates that high-density diffuse optical tomography (HD-DOT), a non-invasive optical neuroimaging technique, can successfully map and decode semantic representations from naturalistic movie viewing. This work bridges the gap between fMRI's detailed semantic mapping capabilities and the accessibility of optical imaging, opening new avenues for studying the brain in more naturalistic settings.

What Problem Does This Paper Address?

Understanding how the brain represents meaning from real-world experiences is crucial for both fundamental neuroscience and clinical applications. While fMRI has been instrumental in creating detailed "semantic maps" of the brain using naturalistic stimuli like movies, its constrained environment limits its use in many real-world scenarios and with certain patient populations. Existing optical neuroimaging techniques like fNIRS have been limited by sparse sampling and simple experimental designs (e.g., block designs), hindering their ability to capture the rich, distributed nature of semantic representations. This paper addresses the challenge of performing high-dimensional semantic encoding and decoding using a more advanced optical imaging method, HD-DOT, in a naturalistic movie-viewing paradigm.

Methods

The core of this study lies in applying advanced machine learning techniques to HD-DOT data. The authors developed a semantic encoding framework where they modeled voxel-level brain activity as a function of semantic features extracted from movies. This involved:

  • High-Density Diffuse Optical Tomography (HD-DOT): A very high-density system with 255 sources and 252 detectors was used, providing dense spatial coverage.
  • L2-Regularized Linear Regression: This statistical method was employed to estimate the relationship between semantic features and brain responses, allowing for the prediction of brain activity from movie content.
  • Finite Impulse Response (FIR) Model: This was used to account for the hemodynamic delay in the brain's response to stimuli.
  • Voxelwise Encoding Model Validation: The accuracy of the encoding models was rigorously tested using held-out data and statistical permutation tests to ensure reliability.
  • Model-Based Semantic Decoding: This involved using the trained encoding models to predict which movie clip a participant was viewing based on their brain activity, assessing the amount of semantic information captured by DOT.
  • Dimensionality Reduction (PCA) and Clustering: Principal Component Analysis was used to identify shared low-dimensional semantic spaces across participants, and k-means clustering was applied to these spaces to uncover interpretable higher-order semantic dimensions.

Datasets

  • Naturalistic Movie Stimuli: The study utilized 3.5 hours of naturalistic movie viewing data per participant. This included 120 minutes of unique movies for training and 9 minutes of repeated movies for testing.
  • Semantic Annotations: The movie stimuli were meticulously annotated with 1,708 semantic categories derived from the WordNet framework, encompassing objects, actions, and their relationships. This rich annotation is a key aspect of the study.
  • Prior fMRI Datasets: The study mentions that the movie stimuli and paradigm were similar to those used in prior fMRI studies, allowing for direct comparison of findings.

Key Insights and Ideas

  • DOT can capture high-dimensional semantic representations: The study demonstrates that HD-DOT, despite being an optical method, can recover complex, distributed semantic information from the brain, similar to what has been achieved with fMRI.
  • Shared semantic space exists across individuals: By using dimensionality reduction techniques, the authors found evidence for a common, low-dimensional semantic space that underlies how different individuals represent meaning from naturalistic stimuli. This suggests a degree of universality in how the brain organizes semantic knowledge.
  • Interpretable semantic dimensions emerge: The analysis revealed distinct, higher-order semantic dimensions (e.g., animate vs. inanimate, natural vs. man-made, textural scenes) that are consistently represented across participants and map onto specific cortical regions. This provides a deeper understanding of the organizational principles of semantic knowledge in the brain.
  • Bridging fMRI and optical imaging: This work successfully bridges the gap between the high spatial resolution of fMRI and the greater accessibility and ecological validity of optical imaging techniques like DOT.

Key Results

  • Robust Semantic Encoding: Encoding models predicted voxel-level brain responses with significant accuracy, yielding semantic category maps consistent with prior fMRI findings.
  • Above-Chance Decoding: DOT responses allowed for the identification of which movie clips participants were viewing, with decoding accuracy remaining robust across various temporal segmentations and training data sizes. For example, 9-way decoding accuracy was 68% ± 11% for 6-way segmentation.
  • Identifiable Semantic Dimensions: PCA and clustering revealed three interpretable semantic dimensions: "Social and Animate Agents," "Objects vs Natural Organisms," and "Textural Scenes." These dimensions showed consistent mapping across participants' cortical surfaces.
  • Cortical Localization: The identified semantic dimensions showed reproducible topographical organization in the cortex, overlapping with known semantic networks identified in fMRI studies.

Limitations and Future Work

  • Depth Limitations of DOT: The authors acknowledge that DOT has inherent depth limitations, meaning it may not capture signals from deeper cortical regions as effectively as fMRI. This is reflected in the less consistent mapping of the "Textural Scenes" cluster.
  • Inter-Subject Variability: While robust findings were observed, the study notes some inter-subject variability in model fit and decoding performance, aligning with previous neuroimaging studies.
  • Future Directions: The authors suggest several promising avenues for future research:
  • Systematic Evaluation of Array Densities: Further research is needed to optimize array configurations for semantic mapping.
  • Multimodal Integration: Combining DOT with other technologies like eye-tracking could provide a more comprehensive understanding of how attention influences semantic representation.
  • Everyday Environments: The development of wearable HD-DOT systems opens possibilities for studying semantic representation in natural, everyday environments.
  • Real-time Semantic Decoding and Neurofeedback: Developing techniques for real-time semantic reconstruction and closed-loop neurofeedback could have significant clinical applications.

Summary updated

Abstract

Understanding how the brain represents meaning in real-world contexts is essential for both fundamental neuroscience and clinical applications. Brain encoding and decoding models from naturalistic stimuli provide a powerful window into semantic representations. Yet, existing approaches rely on a constrained scanning environment, or on conventional fNIRS, which has been limited to sparse sampling and/or block-design paradigms. Here, we tested whether high-density diffuse optical tomography (HD-DOT), an advanced high-density tomographic optical imaging method, can support semantic encoding and decoding using naturalistic movies. We collected 3.5 hours of naturalistic movie viewing data from six participants using stimuli labeled with 1,708 categories. Encoding models robustly predicted voxel-level responses, yielding single semantic category maps consistent with prior fMRI studies. In complementary decoding analyses, we showed that DOT responses captured sufficient semantic content to identify which clips participants viewed. To assess organization across individuals, we identified a shared low-dimensional semantic space that captures common semantic dimensions. Finally, clustering analyses revealed interpretable higher-order semantic dimensions like social and animate agents, objects vs natural organisms, and textural scenes, consistently mapped across the cortex. These findings demonstrate that DOT can recover distributed, high-dimensional semantic representations from naturalistic movies, bridging fMRI-level semantic mapping with the accessibility of optical imaging.
BibTeX
@misc{fehner2025visual,
  author = {Wiete Fehner and Morgan Fogarty and Jerry Tang and Dana Wilhelm and Aahana Bajracharya and Zachary E. Markow and Andrew Hines and Jason W. Trobaugh and Alexander G. Huth and Joseph P. Culver},
  title = {{Visual Semantic Encoding and Identification of Naturalistic Movies via High-Density Diffuse Optical Tomography}},
  year = {2025},
  doi = {10.64898/2025.12.03.692158},
}
Figure