Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN

Tao Fang, Yu Qi, Gang Pan

Abstract

Reconstructing visual stimuli from functional Magnetic Resonance Imaging (fMRI) recordings is a significant challenge in neuroscience, offering a window into brain-reading technology. The primary difficulty lies in the complex, hierarchical nature of visual encoding in the human cortex. This paper introduces a "divide and conquer" strategy implemented via the Shape-Semantic GAN. By decoding low-level structural shapes from the lower visual cortex (LVC) and high-level semantic categories from the higher visual cortex (HVC) separately, and subsequently fusing them through a generative adversarial network, we achieve state-of-the-art reconstruction performance. Our experiments demonstrate that this bio-mimetic approach significantly improves both reconstruction similarity and image quality.

Introduction

The quest to reconstruct what a person is seeing based on their brain activity—often referred to as "mind-reading"—is a cornerstone of modern neuroscience. However, mapping blood-oxygen-level-dependent (BOLD) signals from fMRI to the complex visual world is notoriously difficult. Existing methods often struggle with a trade-off: simple linear models capture basic shapes but fail at natural scenes, while end-to-end deep learning models often produce blurry results that lack structural integrity.

The human brain processes visuals hierarchically: the LVC (areas V1, V2, and V3) handles spatial layout and edges, while the HVC (including the Fusiform Face Area and Lateral Occipital Complex) handles abstract semantics. Most current AI models attempt to decode both simultaneously, leading to suboptimal reconstructions. The Shape-Semantic GAN addresses this by respecting the biological division of labor.

Methodology

The Shape-Semantic GAN framework consists of three distinct stages designed to mirror the brain's hierarchical processing.

The Shape-Semantic GAN framework: (A) Semantic Decoder for HVC features, (B) Shape Decoder for LVC features, and (C) Image Generator for fusion.

1. Shape Decoder

Using fMRI data from the LVC, a linear regression model predicts the binarized outline of the stimulus object. To improve invariance to small distortions and reduce computational overhead, the authors employ a "patch-based" approach, extracting local features to reconstruct the overall spatial structure.

2. Semantic Decoder

Data from the HVC is processed by a lightweight Deep Neural Network (DNN) trained to identify the category of the stimulus image. The features from the penultimate layer of this network are extracted as a "semantic vector," providing the necessary context for the final image generation.

3. Image Generator (GAN)

The fusion stage utilizes a U-Net architecture as the generator. Unlike standard GANs that may lose detail through a narrow bottleneck, the U-Net’s skip connections preserve the structural details provided by the Shape Decoder. The generator takes both the predicted shape and the semantic vector to synthesize a natural-looking image. To mitigate the "small data" problem inherent in fMRI research, the model was pre-trained using data augmentation with 1.2k additional images from ImageNet.

Experimental Results

The model was evaluated using a benchmark dataset consisting of 1,200 training images and 50 test images across three subjects.

Quantitative Performance

The Shape-Semantic GAN achieved a 65.3% pairwise similarity score (using the SSIM metric), outperforming previous leading methods such as Shen et al. (62.9%) and Beliy et al. (64.3%). ROI validation confirmed the hierarchical hypothesis: semantic features were 14.5% more accurately extracted from the HVC than the LVC, while shapes were 19.3% better decoded from the LVC.

Qualitative Analysis

Qualitatively, the reconstructions produced by the Shape-Semantic GAN are significantly more vivid and recognizable than those of previous models. While earlier methods often produced indistinct, monochromatic blobs, our model successfully recovers distinct colors, textures, and structural details that closely match the original stimulus.

Qualitative comparison of image reconstructions. The Shape-Semantic GAN (Ours) produces more vivid and structurally accurate images compared to leading baseline methods.

Discussion and Conclusion

The success of the Shape-Semantic GAN underscores the value of bio-mimicry in AI design. By specializing decoders for different cortical regions, the model overcomes the ambiguity of shape alone—for instance, knowing whether a circular outline should be filled with the texture of a basketball or a clock based on semantic context.

Future work will focus on addressing the challenge of subject specificity, aiming to develop "universal" decoders that can operate across different individuals without subject-specific training. Additionally, as brain-reading technology advances, the researchers emphasize the importance of establishing ethical frameworks to protect cognitive privacy.

Summary updated

Abstract

Reconstructing seeing images from fMRI recordings is an absorbing research area in neuroscience and provides a potential brain-reading technology. The challenge lies in that visual encoding in brain is highly complex and not fully revealed. Inspired by the theory that visual features are hierarchically represented in cortex, we propose to break the complex visual signals into multi-level components and decode each component separately. Specifically, we decode shape and semantic representations from the lower and higher visual cortex respectively, and merge the shape and semantic information to images by a generative adversarial network (Shape-Semantic GAN). This 'divide and conquer' strategy captures visual information more accurately. Experiments demonstrate that Shape-Semantic GAN improves the reconstruction similarity and image quality, and achieves the state-of-the-art image reconstruction performance.
BibTeX
@inproceedings{fang2020reconstructing,
  author = {Tao Fang and Yu Qi and Gang Pan},
  title = {{Reconstructing Perceptive Images from Brain Activity by Shape-Semantic GAN}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=NXdLPxIwy7},
}
Figure