Decoding and Reconstructing Color from Responses in Human Visual Cortex

Gijs Joost Brouwer, David J. Heeger

Abstract

How is color represented by spatially distributed patterns of activity in the visual cortex? Functional magnetic resonance imaging (fMRI) responses to several stimulus colors were analyzed using multivariate techniques: conventional pattern classification, a forward model of idealized color tuning, and principal component analysis (PCA). Stimulus color was accurately decoded from activity in V1, V2, V3, V4, and VO1, but not in LO1, LO2, V3A/B, or MT+. While the conventional classifier and forward model yielded similar accuracies, the forward model uniquely reconstructed novel stimulus colors not used during training. Mean responses across voxels were not reliably distinguishable, suggesting each color is associated with a unique spatially distributed pattern. PCA revealed that in V4 and VO1, the first two principal component scores reflected a progression through perceptual color space. This was not observed in V1, despite it having the highest decoding accuracy, implying a transformation from cone-opponent representations in V1 to a perceptual color space in V4 and VO1.

Introduction

Human color vision begins with three types of cone receptors in the retina, whose signals are processed into "color-opponent" channels. However, the cortical mechanism that transforms these raw signals into the circular "color wheel" of our subjective experience has remained elusive. This study investigates how different regions of the visual cortex represent color, specifically seeking to identify where neural activity begins to reflect perceptual similarity—recognizing, for instance, that orange is more similar to red than to blue.

Methods and Materials

Researchers recorded fMRI activity in five healthy observers with normal color vision. Participants viewed concentric sinusoidal gratings modulating between a neutral gray point and eight distinct locations in the CIE Lab color space, which is designed to be perceptually uniform. These eight colors were equally spaced in hue at a fixed lightness (L) of 75 and a luminance of 8.8 cd/m².

Experimental setup and color space mapping. (A) Eight stimulus colors in CIE L*a*b* space. (B) The same colors in the CIE XYZ 1931 chromaticity diagram. (C) The temporal sequence of stimulus presentation.

Three multivariate approaches were employed to analyze the voxel activity:

  1. Pattern Classification: A "winner-take-all" classifier was used to identify which of the eight colors was being viewed based on spatial patterns of activity.
  2. Forward Modeling: An encoding model characterized each voxel as a weighted sum of six idealized color-tuning channels, allowing for the reconstruction of novel, "unseen" colors.
  3. Principal Component Analysis (PCA): This technique reduced high-dimensional voxel data into a 2D "neural color space" to visualize how each area organizes chromatic information.

Results

Decoding Performance and Regional Specialization

Stimulus color was accurately decoded in early visual areas (V1, V2, V3) and ventral areas (V4, VO1). V1 demonstrated the highest performance, with decoding accuracy reaching approximately 93% when data was aggregated across observers. Conversely, motion-sensitive (MT+) and object-related (LO1, LO2) areas showed accuracies near chance levels, indicating that color information is not uniformly distributed across the visual cortex.

Forward Modeling and Color Reconstruction

The forward model performed comparably to the standard classifier in identifying the eight training colors. However, its true strength lay in its ability to generalize; it successfully predicted brain responses to novel colors excluded from the training set. This suggests that the model captured fundamental properties of cortical color architecture rather than just memorizing a lookup table.

The V1 to V4 Transformation

A critical dissociation was found between decoding accuracy and perceptual representation. While V1 was the most efficient at identifying specific colors, its internal organization (as revealed by PCA) did not match the perceptual color wheel. Instead, V1 activity appeared driven by low-level cone-opponent signals.

In contrast, in areas V4 and VO1, the first two principal components formed a clear, non-self-intersecting circle that mirrored the order of the perceptual color wheel. This indicates a functional transformation: by the time chromatic information reaches V4, the brain has organized it into a space where perceptually similar colors evoke similar neural patterns.

Discussion

These findings demonstrate that the human visual cortex does not simply pass along raw chromatic signals. Instead, there is a clear hierarchy of representation. Early areas like V1 provide high-fidelity decoding based on retinal inputs, but higher-order areas like V4 and VO1 are responsible for constructing the perceptual color space that defines our visual experience. Future research may expand on these findings by exploring variations in saturation and luminance, as well as investigating the neural representation of "unique hues."

Summary updated

Abstract

How is color represented by spatially distributed patterns of activity in visual cortex? Functional magnetic resonance imaging responses to several stimulus colors were analyzed with multivariate techniques: conventional pattern classification, a forward model of idealized color tuning, and principal component analysis (PCA). Stimulus color was accurately decoded from activity in V1, V2, V3, V4, and VO1 but not LO1, LO2, V3A/B, or MT+. The conventional classifier and forward model yielded similar accuracies, but the forward model (unlike the classifier) also reliably reconstructed novel stimulus colors not used to train (specify parameters of) the model. The mean responses, averaged across voxels in each visual area, were not reliably distinguishable for the different stimulus colors. Hence, each stimulus color was associated with a unique spatially distributed pattern of activity, presumably reflecting the color selectivity of cortical neurons. Using PCA, a color space was derived from the covariation, across voxels, in the responses to different colors. In V4 and VO1, the first two principal component scores (main source of variation) of the responses revealed a progression through perceptual color space, with perceptually similar colors evoking the most similar responses. This was not the case for any of the other visual cortical areas, including V1, although decoding was most accurate in V1. This dissociation implies a transformation from the color representation in V1 to reflect perceptual color space in V4 and VO1.
BibTeX
@article{brouwer2009decoding,
  author = {Gijs Joost Brouwer and David J. Heeger},
  title = {{Decoding and Reconstructing Color from Responses in Human Visual Cortex}},
  journal = {Journal of Neuroscience},
  year = {2009},
  doi = {10.1523/jneurosci.3577-09.2009},
}
Figure