Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex

Kuan Han, Haiguang Wen, Junxing Shi, Kun‐Han Lu, Yizhen Zhang, Zhongming LiubioRxiv 2017 · DOI

TL;DR

This paper explores Variational Autoencoders (VAEs) as a computational model for the human visual cortex, aiming to predict and reconstruct brain activity from visual stimuli. While VAEs showed lower prediction accuracy than CNNs, they offered a more direct and effective method for reconstructing visual input from fMRI data, supporting the idea of the brain as a generative model.

What Problem Does This Paper Address?

Understanding how the human visual system processes information is a major challenge. Traditional approaches often rely on supervised learning methods like Convolutional Neural Networks (CNNs) to model visual cortex activity. However, these models are typically trained on labeled data and primarily capture feedforward processing. This paper investigates an alternative approach using Variational Autoencoders (VAEs), which are trained in an unsupervised manner and can model both bottom-up (recognition) and top-down (generation) pathways, aligning more closely with theories of the brain as a generative model. The goal is to see if VAEs can effectively model and decode fMRI responses to naturalistic visual stimuli, and how they compare to established CNN models.

Methods

The core of this research involves training a Variational Autoencoder (VAE) for unsupervised learning of visual representations. The VAE consists of a five-layer encoder and a five-layer decoder.

  • VAE Architecture: The encoder maps input images to a distribution of latent variables, and the decoder reconstructs images from these latent variables. This architecture is inspired by the "free-energy" principle in neuroscience, where the encoder represents bottom-up processing and the decoder represents top-down processing.
  • Training: The VAE was trained on a large dataset of unlabeled natural images to learn compressed, generalized visual representations.
  • Encoding Models: To predict fMRI responses, voxel-wise linear regression models were trained. These models used the VAE's unit activities (after HRF convolution and dimensionality reduction via PCA) as predictors for the fMRI signal.
  • Decoding Models: For visual reconstruction from fMRI, a two-step process was used:
  • A linear regression model transformed fMRI responses into VAE's latent variables.
  • The VAE's decoder then transformed these latent variables into pixel patterns, reconstructing the visual input.
  • Comparison: The VAE-based models were compared against a CNN model (ResNet-18) for encoding performance and against eigen-image-based methods (using PCA and PLSR) for decoding performance.

Datasets

  • Training Data for VAE: The VAE was trained on the ImageNet ILSVRC2012 dataset, which consists of a large collection of natural images. The training data was augmented by horizontal flipping, resulting in over 2 million training samples.
  • fMRI Data: fMRI data was collected from three healthy human volunteers (female, age 23-26) who passively watched natural videos. The data was divided into training and testing sets for model evaluation.

Key Insights and Ideas

  • VAE as a Generative Model: The study strongly supports the idea that the brain, at least in part, operates as a generative model of the visual world. VAEs, with their encoder-decoder structure, naturally embody this generative principle.
  • Unsupervised Learning for Visual Representations: VAEs can learn rich visual representations from unlabeled data, demonstrating the power of unsupervised learning in capturing complex visual features without explicit object labels.
  • VAE for Visual Reconstruction: Despite lower encoding accuracy compared to CNNs, VAEs proved to be a more convenient and effective tool for reconstructing visual input from fMRI data. This suggests that the latent space learned by VAEs captures information that is directly translatable back to visual stimuli.
  • Bridging Neuroscience and AI: The research highlights the conceptual parallels between VAEs and neuroscience theories like the "free-energy" principle, suggesting that these deep learning architectures can serve as valuable computational models for understanding brain function.

Key Results

  • Encoding Performance: VAE-based encoding models achieved statistically significant prediction accuracy in large areas of the visual cortex. Early visual areas (V1/V2/V3) showed the highest accuracy, while higher visual areas had lower prediction accuracy. However, CNNs (ResNet-18) consistently outperformed VAEs in encoding performance across most of the visual cortex, especially in higher visual areas.
  • Decoding Performance: The VAE-based decoding strategy was more advantageous than alternative methods like partial least square regression. It successfully reconstructed both the spatial structure and color of the visual input.
  • The VAE-based decoding method yielded significantly higher Structural Similarity Index (SSIM) scores (around 0.5) compared to eigen-image-based benchmark models.
  • The reconstruction also preserved color information, with statistically significant correlations in hue-value.
  • Cross-Subject Generalization: The VAE-based decoding model showed some degree of transferability across subjects, meaning a model trained on one subject could partially reconstruct visual input from another subject's fMRI data.

Limitations and Future Work

  • Encoding Accuracy: The authors acknowledge that VAE's encoding performance was not as high as CNNs, particularly in higher visual areas. They suggest that supervised learning might be necessary to fully explain ventral-stream activity, which is crucial for object recognition.
  • Hierarchical and Temporal Processing: While VAEs capture bottom-up and top-down processing, they do not explicitly model hierarchical sensory causes or temporal dynamics as effectively as biological brains might. Future work could incorporate these properties into VAE-like models.
  • Blurry Reconstruction: The visual reconstructions, while capturing basic information, were noted to be blurry and lacked fine details.
  • Future Directions: The paper suggests that combining VAEs with other generative models like Generative Adversarial Networks (GANs) could lead to more robust and generalizable visual reconstruction. Further research is also needed to explore various unsupervised learning objectives and their impact on model performance.

Summary updated

Abstract

Abstract Goal-driven and feedforward-only convolutional neural networks (CNN) have been shown to be able to predict and decode cortical responses to natural images or videos. Here, we explored an alternative deep neural network, variational auto-encoder (VAE), as a computational model of the visual cortex. We trained a VAE with a five-layer encoder and a five-layer decoder to learn visual representations from a diverse set of unlabeled images. Inspired by the “free-energy” principle in neuroscience, we modeled the brain’s bottom-up and top-down pathways using the VAE’s encoder and decoder, respectively. Following such conceptual relationships, we used VAE to predict or decode cortical activity observed with functional magnetic resonance imaging (fMRI) from three human subjects passively watching natural videos. Compared to CNN, VAE resulted in relatively lower accuracies for predicting the fMRI responses to the video stimuli, especially for higher-order ventral visual areas. However, VAE offered a more convenient strategy for decoding the fMRI activity to reconstruct the video input, by first converting the fMRI activity to the VAE’s latent variables, and then converting the latent variables to the reconstructed video frames through the VAE’s decoder. This strategy was more advantageous than alternative decoding methods, e.g. partial least square regression, by reconstructing both the spatial structure and color of the visual input. Findings from this study support the notion that the brain, at least in part, bears a generative model of the visual world.
BibTeX
@misc{han2017variational,
  author = {Kuan Han and Haiguang Wen and Junxing Shi and Kun‐Han Lu and Yizhen Zhang and Zhongming Liu},
  title = {{Variational Autoencoder: An Unsupervised Model for Modeling and Decoding fMRI Activity in Visual Cortex}},
  year = {2017},
  doi = {10.1101/214247},
  howpublished = {bioRxiv},
}

Loading related content...

Figure