Brain Decoding of Multiple Subjects for Estimating Visual Information Based on a Probabilistic Generative Model

Takaaki Higashi, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper introduces a novel brain decoding method that leverages both shared and individual brain activity patterns across multiple subjects to improve the accuracy of estimating visual information. By employing a probabilistic generative model, the approach effectively disentangles common cognitive signals from subject-specific nuances, leading to superior performance in visual cognition decoding compared to existing methods.

What Problem Does This Paper Address?

Decoding human cognitive states from brain activity, particularly visual perception, is a challenging task. This difficulty stems from the inherent complexity of the brain, characterized by high-dimensional data and limited sample sizes for individual subjects. Existing brain decoding methods often struggle to achieve high accuracy, especially when dealing with data from multiple individuals. While some approaches attempt to extract commonalities across subjects, they often overlook the crucial individual differences in brain responses. This paper aims to bridge this gap by proposing a method that explicitly models and utilizes both shared and individual brain activity patterns to enhance the accuracy of estimating visual information.

Methods

The core of the proposed method lies in a two-step approach using a probabilistic generative model (PGM).

  1. PGM for Feature Extraction: A PGM is constructed to disentangle "shared features" (common across subjects) and "individual features" (unique to each subject) from fMRI data. This model is designed to be robust to noise in fMRI signals and can effectively learn from limited data. The PGM assumes Gaussian distributions for these features, allowing for analytical calculation of expected values and variances. The expectation-maximization (EM) algorithm is used to update the model parameters.
  2. Visual Decoder Construction: A visual decoder is then trained to estimate visual features from the extracted shared and individual features. This decoder uses projection matrices to map the latent features into a visual representation. The training of the decoder involves minimizing the difference between the estimated visual features and those derived from a pre-trained CNN (VGG19).

The key novelty here is the explicit separation and joint utilization of shared and individual features, which are then fed into a decoder for visual estimation.

Datasets

The study utilized a publicly available fMRI dataset.

  • Dataset: fMRI data from five subjects who viewed images. The dataset comprised approximately 4500-dimensional fMRI vectors.
  • Image Data: 1200 seen images from 150 categories in ImageNet were used.
  • Data Split: The dataset was divided into training (900 pairs), validation (150 pairs), and test (150 pairs) sets.
  • Cross-validation: 7-fold cross-validation was applied to 1050 pairs of training and test data.

Key Insights and Ideas

  • The Power of Disentanglement: Explicitly separating shared and individual brain activity patterns is crucial for improving brain decoding accuracy. Neither shared nor individual information alone is sufficient; their combination yields superior results.
  • Generative Models for Limited Data: Probabilistic generative models are effective for learning from fMRI data, which is often limited due to the burden on subjects. They can extract robust features even with small sample sizes.
  • Bridging Brain and Vision: The method successfully links brain activity patterns to visual features, demonstrating that the brain's representation of visual information can be decoded by understanding both common and unique neural responses.
  • Limitations of Deep Learning for fMRI: The paper suggests that deep learning methods, while powerful, might not be as compatible with fMRI data due to its inherent noise and limited sample size, as evidenced by the proposed method outperforming Deep CCA.

Key Results

The proposed method (PM) demonstrated superior performance in estimating visual information compared to several baseline methods.

  • Quantitative Evaluation: The PM achieved an average confidence category score of 0.813 across five subjects, with the best subject scoring 0.867. This outperformed all comparison methods, including multi-subject approaches like MSPGM (average 0.805) and MVBGM-MS (average 0.792), and single-subject methods like SSPGM (average 0.794) and SLR (average 0.769).
  • Qualitative Evaluation: In qualitative assessments for image categories like "shirt" and "saddle," the PM consistently achieved high confidence scores across subjects, indicating its robustness. For categories like "hand calculator" and "obelisk," the PM also showed strong performance, though some comparison methods excelled in specific instances, highlighting the nuances of feature extraction.

Limitations and Future Work

  • Limitations:
    • The study acknowledges that fMRI data inherently contains noise and significant individual differences, which can still pose challenges.
  • The performance on images with multiple objects or objects not directly related to the category was less sufficient, suggesting that gaze focus and object complexity can impact decoding accuracy.
  • Future Work:
    • Increasing Sample Size: Expanding the fMRI dataset and incorporating more subjects is a natural next step.
  • Multimodal Integration: The authors propose integrating other modalities, such as image captions, to enrich the visual information beyond what CNNs can extract (e.g., color and shape). This could further improve the expressive ability of the decoded features.
  • Exploring Caption Features: Investigating the use of image captioning models to extract features that complement fMRI data is a promising direction.

Summary updated

Abstract

Brain decoding is a process of decoding human cognitive contents from brain activities. However, improving the accuracy of brain decoding remains difficult due to the unique characteristics of the brain, such as the small sample size and high dimensionality of brain activities. Therefore, this paper proposes a method that effectively uses multi-subject brain activities to improve brain decoding accuracy. Specifically, we distinguish between the shared information common to multi-subject brain activities and the individual information based on each subject’s brain activities, and both types of information are used to decode human visual cognition. Both types of information are extracted as features belonging to a latent space using a probabilistic generative model. In the experiment, an publicly available dataset and five subjects were used, and the estimation accuracy was validated on the basis of a confidence score ranging from 0 to 1, and a large value indicates superiority. The proposed method achieved a confidence score of 0.867 for the best subject and an average of 0.813 for the five subjects, which was the best compared to other methods. The experimental results show that the proposed method can accurately decode visual cognition compared with other existing methods in which the shared information is not distinguished from the individual information.
BibTeX
@inproceedings{higashi2022brain,
  author = {Takaaki Higashi and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Brain Decoding of Multiple Subjects for Estimating Visual Information Based on a Probabilistic Generative Model}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=sEipruVDd22},
}
Figure