Abstract
Addressing the question of visualising human mind could help us to find regions that are associated with observed cognition and responsible for expressing the elusive mental image, leading to a better understanding of cognitive function. The traditional approach treats brain decoding as a classification problem, reading the mind through statistical analysis of brain activity. However, human thought is rich and varied, that it is often influenced by more of a combination of object features than a specific type of category. For this reason, we propose an end-to-end brain decoding framework which translates brain activity into an image by latent space alignment. To find the correspondence from brain signal features to image features, we embedded them into two latent spaces with modality-specific encoders and then aligned the two spaces by minimising the distance between paired latent representations. The proposed framework was trained by simultaneous electroencephalogram and functional MRI data, which were recorded when the subjects were viewing or imagining a set of image stimuli. In this paper, we focused on implementing the fMRI experiment. Our experimental results demonstrated the feasibility of translating brain activity to an image. The reconstructed image matches image stimuli approximate in both shape and colour. Our framework provides a promising direction for building a direct visualisation to reveal human mind.
Introduction: From Classification to Translation
Can we actually "see" what someone else is thinking? For decades, brain decoding—the art of reading mental states from neural activity—has largely been treated as a classification problem. Researchers could tell if you were looking at a cat or a house, but they couldn't show you the specific cat your mind was picturing.
A new framework proposes a shift in perspective: Decoding as Translation. Instead of labeling brain activity, this system attempts to translate it directly into images using a sophisticated generative adversarial network (GAN) architecture. By moving beyond simple classification to "latent space translation," the system can generate images that approximate the shape and color of what a subject is actually seeing.
The Challenge of Brain Decoding
Traditional brain decoding relies on statistical analysis to categorize brain activity into discrete buckets (e.g., "Face" vs. "Object"). However, human thought isn't discrete; it’s a rich, fluid combination of features like texture, color, and geometry.
The challenge is that brain signals (like fMRI voxels or EEG waves) and digital images speak completely different "languages." To reconstruct an image, a bridge is needed to map high-dimensional, noisy brain data onto structured pixel data. Previous attempts often resulted in blurry, unrecognizable blobs or were limited to very specific, pre-trained categories.
Methodology: The $\alpha$-GAN Framework
The proposed "Decoding as Translation" framework is centered around an $\alpha$-GAN (Alpha-GAN) and operates in three distinct stages.
- Image Feature Learning: First, an $\alpha$-GAN is trained on a large dataset of images. An $\alpha$-GAN combines a Variational Auto-encoder (VAE) with a GAN. The VAE ensures the latent space is well-structured, while the GAN’s adversarial loss prevents the reconstructed images from being blurry—a common failing of standard VAEs.
- Brain Signal Encoding: This stage acts as the "bridge." Bayesian regression models map specific features of brain activity (fMRI or EEG) to the 128-dimensional latent vector ($z$) used by the image generator. Pearson Correlation Coefficients are used to select the most relevant brain "features" (voxels or channels) for each dimension of the latent vector.
- The Translation: Finally, the trained brain encoder is combined with the pre-trained image generator. When a subject sees an image, their brain activity is encoded into a latent vector, which the GAN then "translates" back into a reconstructed image.
Experimental Setup and Datasets
The study utilized a specialized dataset of simultaneous EEG and fMRI recordings from 5 healthy volunteers. The stimuli consisted of 1,000 images across 20 categories (e.g., backpacks, butterflies, jellyfish). Sessions included both "image presentation" (viewing) and "mental imagery" (imagining). While many studies use one modality, this framework is designed to be "modality-agnostic," meaning it can process either EEG or fMRI signals to perform the reconstruction.
Key Insights
- Latent Space Alignment: The core innovation is that direct pixel-to-voxel mapping is unnecessary. By mapping both to a shared "latent representation," the generative model handles the complex task of rendering the image.
- The $\alpha$-GAN Advantage: By using $\alpha$-GAN instead of a standard GAN or VAE, the researchers addressed "model collapse" (where GANs stop producing variety) and "blurriness" (where VAEs produce fuzzy results).
- fMRI vs. EEG Trade-offs: fMRI provides excellent spatial resolution, making it better for reconstructing shapes. EEG provides better temporal resolution but is noisier, making it sufficient for classification but difficult for high-fidelity reconstruction.
Results and Discussion
The researchers evaluated results using the Structural Similarity Index (SSIM) and Mean-Square-Error (MSE).
- fMRI Success: fMRI-based reconstructions achieved a "correct rate" of approximately 80%. Generated images clearly matched stimuli in primary colors and rough geometric shapes.
- EEG Limitations: The EEG framework achieved a correct rate of over 60%, but resulting images were blurrier and harder to distinguish by shape alone.
- Feasibility: The study proved that a single framework can successfully "translate" neural activity into a recognizable visual format.
Conclusion and Future Work
While we aren't yet at the level of recording dreams, this framework provides a promising direction for direct visualization of the human mind. Current limitations include image resolution—constrained by fMRI voxel size—and the high noise floor of EEG. Future work will focus on improving fine-grained detail and enhancing the reconstruction of purely internal, imagined thoughts.
Reference: Wang, P., Zhou, R., Wang, S., et al. (2021). A General Framework for Revealing Human Mind with auto-encoding GANs. arXiv:2102.05236v1 [q-bio.NC].
Summary updated