Inter-individual deep image reconstruction

Jun Kai Ho, Tomoyasu Horikawa, Kei Majima, Fan Cheng, Yukiyasu KamitanibioRxiv 2022 · DOI

TL;DR

This paper introduces a "neural code converter" that translates brain activity patterns between individuals, enabling the reconstruction of visual images from one person's brain activity for another. This method successfully preserves hierarchical visual information, demonstrating a significant step towards cross-individual brain decoding.

What Problem Does This Paper Address?

The human brain, particularly the visual cortex, processes information hierarchically, starting with simple features and building up to complex perceptions. However, even when presented with the same visual input, individuals exhibit distinct patterns of brain activity. While methods exist to align brains anatomically or functionally, it's unclear if and how the detailed, hierarchical representations of visual information can be reliably transferred between individuals to reconstruct what they see. This paper tackles the challenge of bridging this inter-individual gap in brain activity to enable cross-subject visual perception reconstruction.

Methods

The core of this research is the neural code converter, a machine learning model trained to predict a target subject's brain activity from a source subject's activity, given the same visual stimulus. This is achieved using regularized linear regression (ridge regression). Crucially, the converter is trained without explicit labels of visual areas or their hierarchy.

The study also employs:

  • DNN feature decoding: Using pre-trained deep neural networks (DNNs) to decode visual features from the converted brain activity.
  • Deep image reconstruction: A method to reconstruct images from these decoded DNN features.
  • Ablation analysis: Systematically removing parts of the source brain activity to understand which areas contribute most to successful conversion.
  • Various functional alignment methods: Comparing the neural code converter to other techniques like Procrustes transformation, optimal transport, and hyperalignment to assess robustness.

Datasets

The study utilizes fMRI data from human subjects viewing natural and artificial images. Specifically:

  • Training data: 6,000 samples of fMRI data per subject for natural images.
  • Test data: 1,200 samples of natural images and 800 samples of artificial images per subject.
  • Data was collected from five subjects, with some data reused from previous studies (Shen et al., 2019a; Horikawa and Kamitani, 2022).

Key Insights and Ideas

  • Automatic Hierarchical Learning: The neural code converters learn the correspondence between visual areas across individuals without being explicitly told about the brain's hierarchical structure. This suggests that the underlying organizational principles are implicitly captured.
  • Preservation of Hierarchical Representations: Decoding DNN features from converted brain activity showed higher accuracy at corresponding hierarchical levels of visual areas, indicating that the fine-grained, hierarchical nature of visual information is preserved during conversion.
  • Reconstruction Feasibility with Limited Data: Even with a small number of training samples (as few as 300), the converters could produce recognizable visual reconstructions, highlighting the efficiency of this inter-individual approach.
  • Whole VC vs. Subarea-wise: While explicit information about visual subareas didn't significantly improve reconstruction with sufficient data, the whole visual cortex (VC) conversion showed slightly better performance, suggesting that a broader mapping might be more flexible.

Key Results

  • Moderate Conversion Accuracy: The neural code converters achieved moderate conversion accuracy, with mean normalized pattern correlation of 0.56 and profile correlation of 0.53 for the whole VC across 20 individual pairs.
  • Recognizable Reconstructions: Reconstructed images, even from converted brain activity, showed recognizable shapes and colors of objects.
  • Improved Reconstruction with Pooling: Pooling data from multiple subjects led to a slight improvement in accuracy for inter-individual decoding and reconstruction, though visual quality didn't significantly increase.
  • Subarea-wise Conversion Performance: Subarea-wise conversion showed slightly lower decoding accuracy and identification accuracy compared to whole VC conversion, especially with sufficient training data.

Limitations and Future Work

  • Moderate Accuracy: While functional alignment works, the conversion accuracy is described as "moderate." More advanced methods might be needed to fully resolve feature mismatches between individuals.
  • Pooling Data Benefits: Pooling data showed only slight improvements, possibly due to variability in data quality and the limitations of linear regression models in fully capturing complex relationships.
  • Linear Constraint: The linear constraint used in conversion might be too restrictive for learning more complex, non-linear relationships between brain activity patterns.
  • Future Directions: The authors suggest that this approach offers an efficient way to reconstruct visual images without needing extensive subject-specific data, potentially popularizing brain-machine interfaces. Further research could explore more advanced pooling strategies and non-linear models.

Summary updated

Abstract

Abstract The sensory cortex is characterized by general organizational principles such as topography and hierarchy. However, measured brain activity given identical input exhibits substantially different patterns across individuals. Although anatomical and functional alignment methods have been proposed in functional magnetic resonance imaging (fMRI) studies, it remains unclear whether and how hierarchical and fine-grained representations can be converted between individuals while preserving the encoded perceptual content. In this study, we trained a method of functional alignment called neural code converter that predicts a target subject’s brain activity pattern from a source subject given the same stimulus, and analyzed the converted patterns by decoding hierarchical visual features and reconstructing perceived images. The converters were trained on fMRI responses to identical sets of natural images presented to pairs of individuals, using the voxels on the visual cortex that covers from V1 through the ventral object areas without explicit labels of the visual areas. We decoded the converted brain activity patterns into the hierarchical visual features of a deep neural network using decoders pre-trained on the target subject and then reconstructed images via the decoded features. Without explicit information about the visual cortical hierarchy, the converters automatically learned the correspondence between visual areas of the same levels. Deep neural network feature decoding at each layer showed higher decoding accuracies from corresponding levels of visual areas, indicating that hierarchical representations were preserved after conversion. The visual images were reconstructed with recognizable silhouettes of objects even with relatively small numbers of data for converter training. The decoders trained on pooled data from multiple individuals through conversions lead to a slight improvement over those trained on a single individual. These results demonstrate that the hierarchical and fine-grained representation can be converted by functional alignment, while preserving sufficient visual information to enable inter-individual visual image reconstruction. Highlights ● Neural code converters convert brain activity patterns across individuals with moderate conversion accuracy and learn reasonable visual area correspondences at the same level between individuals. ● The converted brain activity patterns can be decoded into hierarchical DNN features to reconstruct visual images, even though the converter is trained on a small number of data samples. ● The information of hierarchical and fine-scale visual features that enable visual image reconstruction are preserved after functional alignment.
BibTeX
@misc{ho2022inter,
  author = {Jun Kai Ho and Tomoyasu Horikawa and Kei Majima and Fan Cheng and Yukiyasu Kamitani},
  title = {{Inter-individual deep image reconstruction}},
  year = {2022},
  doi = {10.1101/2021.12.31.474501},
  howpublished = {bioRxiv},
}
Figure