Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding

Yusuke Akamatsu, Keisuke Maeda, Takahiro Ogawa, Miki Haseyama

TL;DR

This paper tackles the challenge of "zero-shot neural decoding," where we try to understand what someone is seeing from their brain activity (fMRI) for categories they haven't been trained on. They propose a novel approach using "semi-supervised multi-view embedding" that leverages additional images and combines visual and semantic information to overcome a key problem: the "projection domain shift." This method significantly improves decoding accuracy for unseen categories.

What Problem Does This Paper Address?

Imagine trying to figure out what image a person is looking at, but you only have brain activity data (fMRI) for a limited set of categories (e.g., cats, dogs, cars). Now, you want to decode images from categories you haven't trained on (e.g., a specific type of bird or a rare artifact). This is the core of "zero-shot neural decoding." A major hurdle is that fMRI data is hard and expensive to collect, meaning we have very little training data for many categories. This scarcity leads to a "projection domain shift" problem: the way the model learns to map brain activity to visual features from the training categories doesn't accurately translate to the unseen target categories. This results in poor decoding performance for novel items. The paper aims to bridge this gap by developing a method that can decode unseen categories more effectively, despite limited fMRI training data.

Methods

The authors introduce a "semi-supervised multi-view generative model." Here's a breakdown:

  • Semi-Supervised Learning: They leverage "additional images" related to the target categories, even if these images don't have corresponding fMRI data. This is crucial because collecting fMRI data is the bottleneck. By using these extra images, they can "compensate" for the lack of direct fMRI training data for novel categories.
  • Multi-View Embedding: Instead of relying on a single type of information, their model projects fMRI activity into a combined "embedding space." This space includes:
  • Visual Features: Extracted from images using a Convolutional Neural Network (CNN).
  • Semantic Features: Derived from image categories using a word embedding model (like word2vec). By combining these views, the model can exploit complementary information, leading to a more robust representation.

  • Generative Model with Bayesian Inference: The core of their approach is a generative model that assumes fMRI activity, visual features, and semantic features are generated from a shared latent variable. They use Bayesian inference and Automatic Relevance Determination (ARD) to handle the high dimensionality of fMRI data and automatically select important components, preventing overfitting.

  • Addressing Projection Domain Shift: The semi-supervised component, by incorporating additional images, and the multi-view embedding, by combining visual and semantic information, work together to alleviate the projection domain shift problem. This means the learned mappings are more generalizable to unseen categories.

Datasets

The study utilizes a publicly available fMRI dataset [6] which includes:

  • Training Data: fMRI activity patterns from five subjects viewing 1200 images across 150 categories (8 images per category).
  • Test Data: fMRI activity patterns from the same subjects viewing 50 images from 50 new categories (not seen during training), with each image presented 35 times.
  • Additional Data: For their semi-supervised approach, they incorporate additional image categories from ImageNet, which are used to extract visual and semantic features but do not require corresponding fMRI data.

Key Insights and Ideas

  • Leveraging Unlabeled Images is Key: The most significant insight is that using additional images without fMRI data for target categories can dramatically improve zero-shot decoding performance. This bypasses the fMRI data collection bottleneck.
  • Visual + Semantic is Better Than Either Alone: Combining visual and semantic features in a multi-view embedding space is more effective than using either alone, especially when dealing with diverse categories. The optimal balance between visual and semantic features can vary depending on the category group.
  • Projection Domain Shift is a Real Problem: The paper clearly demonstrates and quantifies the "projection domain shift" problem, showing how it biases decoding towards source categories and hinders performance on unseen ones. Their method directly addresses this.
  • ARD for fMRI Data: The use of Automatic Relevance Determination (ARD) in their Bayesian model is crucial for handling the high dimensionality of fMRI data, allowing the model to focus on relevant voxels and improve generalization.

Key Results

  • Outperforms Existing Methods: The proposed semi-supervised multi-view embedding approach significantly outperforms seven other methods, including existing neural decoding techniques (SLR, CCA, Semi-FDCCA, MLP), BCCA, and a multi-view generative model (MGM) without the semi-supervised component.
  • Improved Identification Accuracy: In a zero-shot setting with 10,000 random candidate categories, their method achieved the highest identification accuracy across various target groups, with results like 91.63% for commodity and 93.10% for mammal.
  • Better Performance with Similar Categories: Even when identifying from similar categories within the same target group (a more challenging task), their model still showed superior performance compared to others.
  • Visual vs. Semantic Importance Varies: The study found that for "living thing" categories, semantic features were more important in the zero-shot setting, while for "artifact" categories, visual features were more dominant. This highlights the need for a balanced multi-view approach.

Limitations and Future Work

  • Data Scarcity: The fundamental limitation remains the difficulty and expense of collecting fMRI data, which is the driving force behind the zero-shot approach.
  • Generalizability of Additional Categories: While using additional categories is beneficial, the paper notes that using categories similar to the target group yields better results than unrelated ones. Future work could explore more sophisticated ways to select or generate these additional features.
  • Trade-off Parameter Tuning: The optimal balance (η) between visual and semantic features needs to be determined, and while they explored this, further research could automate or refine this process.
  • Broader Applicability: The authors suggest their framework could be applied to other challenging learning tasks where one modality is abundant (like images) and another is scarce (like fMRI).

Summary updated

Abstract

Zero-shot neural decoding aims to decode image categories, which were not previously trained, from functional magnetic resonance imaging (fMRI) activity evoked when a person views images. However, having insufficient training data due to the difficulty in collecting fMRI data causes poor generalization capability. Thus, models suffer from the projection domain shift problem when novel target categories are decoded. In this paper, we propose a zero-shot neural decoding approach with semi-supervised multi-view embedding. We introduce the semi-supervised approach that utilizes additional images related to the target categories without fMRI activity patterns. Furthermore, we project fMRI activity patterns into a multi-view embedding space, i.e., visual and semantic feature spaces of viewed images to effectively exploit the complementary information. We define several source and target groups whose image categories are very different and verify the zero-shot neural decoding performance. The experimental results demonstrate that the proposed approach rectifies the projection domain shift problem and outperforms existing methods.
BibTeX
@inproceedings{akamatsu2023zero,
  author = {Yusuke Akamatsu and Keisuke Maeda and Takahiro Ogawa and Miki Haseyama},
  title = {{Zero-Shot Neural Decoding with Semi-Supervised Multi-View Embedding}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=9z0uV0uJdQ},
}
Figure