Conditional Generative Neural Decoding with Structured CNN Feature Prediction

Changde Du, Changying Du, Lijie Huang, Huiguang He

Abstract

Decoding visual stimuli from human brain activity (fMRI) is a fundamental challenge in neuroscience. This paper presents a novel two-stage framework that addresses data scarcity and the neglect of structural dependencies in neural signals. The first stage, Voxel2Unit, employs Structured Multi-output Regression (SMR) to predict CNN features by modeling correlations between voxels and feature units. The second stage, Unit2Pixel, utilizes Introspective Conditional Generation (ICG) to reconstruct high-fidelity images. By leveraging large-scale unpaired image datasets and an introspective learning objective, the proposed method achieves state-of-the-art performance in reconstructing both natural scenes and human faces.

Introduction

Decoding visual content from functional Magnetic Resonance Imaging (fMRI) is often considered a "holy grail" of neuroscience and AI. However, two major hurdles have historically limited progress. First, data scarcity: collecting paired data (where a person looks at an image while their brain is scanned) is expensive and slow, resulting in small datasets. Second, ignoring structure: most existing methods treat brain voxels and neural network features as independent units. In reality, brain regions are highly interconnected, and the features of a Convolutional Neural Network (CNN) are deeply correlated.

Overview of the proposed neural decoding pipeline, from visual stimulus to reconstructed image.

Traditional methods often produce blurry, unrecognizable reconstructions because they fail to exploit these underlying structures or lack the generative "imagination" to fill in missing details. This paper introduces a framework that explicitly models these structures and uses generative priors to produce sharp, semantically accurate images.

Methods

The authors propose a two-stage pipeline: Voxel2Unit and Unit2Pixel.

1. Voxel2Unit: Structured Multi-output Regression (SMR)

Instead of predicting CNN features independently, the SMR model predicts them simultaneously. It uses a Bayesian approach to capture three types of structure: input structure (correlations between fMRI voxels), output structure (dependencies between CNN units), and task structure (relationships between individual regression tasks). By learning sparse inverse covariance matrices, the model ignores noise and focuses on meaningful functional networks in the brain.

2. Unit2Pixel: Introspective Conditional Generation (ICG)

Once CNN features are predicted, they are transformed back into an image using the ICG model, an "introspective" Variational Autoencoder (VAE). Unlike standard GANs, the ICG's own encoder acts as a discriminator, evaluating the quality of generated images by measuring their deviation from the expected data distribution via KL divergence. This approach avoids the training instability of GANs while achieving high-frequency detail. Crucially, the ICG can be pre-trained on massive external datasets (like ImageNet or CelebA) to learn the "language of vision" before being conditioned on fMRI-derived features.

Key Results

The framework was evaluated on the Vim-1 dataset (natural images) and a new FaceBold dataset (human faces). Quantitatively, the SMR model outperformed standard linear regressions (SLR) and Bayesian Canonical Correlation Analysis (BCCA) in feature prediction accuracy. On the Vim-1 dataset, the method achieved a Pearson Correlation Coefficient (PCC) of 0.551 and a Structural Similarity Index (SSIM) of 0.675.

Qualitative comparison of image reconstructions across different methods for natural scenes and faces.

Qualitatively, the images produced by the ICG model are significantly sharper and more semantically accurate than those from standard VAEs or de-convolutional networks. In the FaceBold dataset, the reconstructions are detailed enough to resemble specific individuals rather than generic face-shaped blobs.

Discussion and Future Work

By treating structure as a signal rather than noise and leveraging "unpaired" images to bridge the data gap, this work sets a new benchmark for neural decoding. Future directions include scaling the model to handle more CNN units, extending the framework to high-resolution color images, and incorporating cycle consistency to further refine reconstruction accuracy.

Summary updated

Abstract

Decoding visual contents from human brain activity is a challenging task with great scientific value. Two main facts that hinder existing methods from producing satisfactory results are 1) typically small paired training data; 2) under-exploitation of the structural information underlying the data. In this paper, we present a novel conditional deep generative neural decoding approach with structured intermediate feature prediction. Specifically, our approach first decodes the brain activity to the multilayer intermediate features of a pretrained convolutional neural network (CNN) with a structured multi-output regression (SMR) model, and then inverts the decoded CNN features to the visual images with an introspective conditional generation (ICG) model. The proposed SMR model can simultaneously leverage the covariance structures underlying the brain activities, the CNN features and the prediction tasks to improve the decoding accuracy and interpretability. Further, our ICG model can 1) leverage abundant unpaired images to augment the training data; 2) self-evaluate the quality of its conditionally generated images; and 3) adversarially improve itself without extra discriminator. Experimental results show that our approach yields state-of-the-art visual reconstructions from brain activities.
BibTeX
@inproceedings{du2020conditional,
  author = {Changde Du and Changying Du and Lijie Huang and Huiguang He},
  title = {{Conditional Generative Neural Decoding with Structured CNN Feature Prediction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5647},
}
Figure