Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

Jiaxuan Chen, Yu Qi, Gang PanICML 2023 · OpenReview

Reconstructing what a person is seeing based on functional Magnetic Resonance Imaging (fMRI) data is a "holy grail" of neural decoding. However, existing methods often produce blurry, unrecognizable blobs or "hallucinate" details that fail to match the original stimulus.

Researchers have developed VQ-fMRI, a new framework that reconstructs images from brain activity by mimicking the human brain's ability to "fill in the blanks." By using discrete visual tokens and a self-supervised inpainting network, the system moves beyond simple pixel-mapping to produce state-of-the-art reconstructions that are both visually coherent and semantically accurate.

The Biological Inspiration: Brain Completion

The authors argue that the core issue with previous reconstruction attempts is a fundamental misunderstanding of human biology. Our visual system isn't a high-resolution camera; it is a lossy processor. We use selective attention to focus on specific parts of a scene, and our brains use prior experience to "guess" the rest.

The human visual system relies on selective attention and brain completion to form perceived images.

Most AI models try to map noisy fMRI signals directly to every pixel of an image, which is akin to reconstructing a 4K movie from a handful of static-filled radio pips. This paper proposes a paradigm shift: instead of direct pixel-by-pixel reconstruction, the model should perform experience-based content completion.

The VQ-fMRI Framework

To achieve this, the authors introduce VQ-fMRI, which breaks the reconstruction process into distinct stages designed to filter noise and leverage learned priors.

The VQ-fMRI architecture, featuring a shared discrete space and a token-to-token inpainting network.
  1. Learning a Visual Vocabulary (VQ-VAE): Instead of working with continuous pixels, the model uses a Vector-Quantized Variational Autoencoder to learn a "codebook" of discrete visual tokens. Think of these as the "atoms" of an image, representing textures, shapes, and parts of objects.
  2. Mapping Brain Signals to Tokens: A specialized encoder maps fMRI voxels to these discrete tokens. Because this space is discrete, the model is significantly more robust to the inherent noise in fMRI data. It simply needs to find the "closest match" in the codebook rather than predicting an exact, continuous pixel value.
  3. Token-to-Token Inpainting: This is the "brain completion" step. Since fMRI signals are often incomplete, the model uses a self-supervised inpainting network (trained on ImageNet) to predict missing or corrupted tokens based on the surrounding context and "prior knowledge" of what objects usually look like.
  4. Hierarchical Super-Resolution: To avoid overfitting to high-frequency noise, the model first reconstructs a low-resolution "outline" and then uses a hierarchical U-Net structure to add details, ensuring the global layout remains consistent with the original stimulus.

Key Insights and Performance

The most novel aspect of this research is treating visual reconstruction as an image restoration task. The model acknowledges that the fMRI signal is "broken" or "incomplete" and uses a learned prior to fix it, much like how our own visual cortex fills in the blind spot in our eyes. Furthermore, by forcing the fMRI-to-image mapping into a discrete latent space, the authors effectively "quantize" away the noise. Small fluctuations in fMRI signals no longer cause massive pixel distortions.

Tested on the Generic Object Decoding (GOD) Dataset, the VQ-fMRI method achieved significant quantitative gains over previous state-of-the-art methods (like IC-GAN and various encoder-decoder models):

  • Structural Similarity (SSIM): Achieved 0.492, an 18% to 32% improvement over competitors, indicating much better preservation of local spatial structures.
  • Peak Signal-to-Noise Ratio (PSNR): Reached 13.4 dB, outperforming others by 9% to 33%.
  • Pearson Correlation Coefficient (PCC): Achieved 0.551, a massive jump (up to 128% higher than some GAN-based methods), proving the reconstructions are much more linearly correlated with the original stimuli.

Visually, the reconstructions show clear backgrounds, recognizable silhouettes, and consistent color palettes where previous methods produced gray or distorted textures.

Limitations and Future Work

While impressive, the model still encounters "bad cases." For instance, if the discrete cues decoded from the fMRI are fundamentally ambiguous (e.g., a "cup" being misidentified as a "bell" at the token level), the inpainting module will confidently complete the image based on that incorrect cue.

The authors note that their current inpainting uses a relatively simple approach. They suggest that integrating Diffusion Models—currently the gold standard for image generation—into their "token-to-token" framework could produce even more photorealistic and semantically complex results in the future. Finally, the principle of "experience-based completion" could potentially be applied to other neural decoding fields, such as reconstructing audio from brain activity.

Summary updated

Abstract

Decoding seen images from brain activities has been an absorbing field. However, the reconstructed images still suffer from low quality with existing studies. This can be because our visual system is not like a camera that ''remembers'' every pixel. Instead, only part of the information can be perceived with our selective attention, and the brain ''guesses'' the rest to form what we think we see. Most existing approaches ignored the brain completion mechanism. In this work, we propose to reconstruct seen images with both the visual perception and the brain completion process, and design a simple, yet effective visual decoding framework to achieve this goal. Specifically, we first construct a shared discrete representation space for both brain signals and images. Then, a novel self-supervised token-to-token inpainting network is designed to implement visual content completion by building context and prior knowledge about the visual objects from the discrete latent space. Our approach improved the quality of visual reconstruction significantly and achieved state-of-the-art.
BibTeX
@inproceedings{chen2023rethinking,
  author = {Jiaxuan Chen and Yu Qi and Gang Pan},
  title = {{Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=l3sdNQdmQh},
}
Figure