A Computational Framework for Modeling Emergence of Color Vision in the Human Brain

Atsunobu Kotani, Ren Ng

Introduction: The Challenge of Neural Color Decoding

Human color vision is often perceived as a direct mapping of the external world, yet from a signal-processing standpoint, it is remarkably complex. The optic nerve signals (ONS) transmitted to the brain are not clean "images" but rather a noisy, spatially warped mosaic of activations from different cone cell types (L, M, and S). These signals are further distorted by the eye's lens and constant, involuntary fixational eye drift.

A fundamental question in neuroscience is how the brain decodes this "mess" into a coherent, multi-dimensional color space without prior knowledge of the eye's physical wiring or access to ground-truth "teacher" data. This research presents a computational framework that demonstrates how color vision can emerge purely through self-supervised learning.

Methodology: Simulating the Visual Pathway

The researchers developed a two-stage system to model the emergence of color vision, moving from raw hyperspectral input to a decoded internal representation.

Overview of the computational framework, detailing the transition from hyperspectral scene stimulus to emergent neural color analysis.

1. The Simulation Engine of Biological Eyes

This component acts as a biophysically accurate retina model. It processes hyperspectral images—which contain full spectral power distributions—and converts them into realistic ONS. The model accounts for foveation (the non-uniform "fish-eye" spatial warping), patchy spectral sampling (the random mosaic arrangement of different cone types), and fixational eye drift (tiny, constant movements of the eye).

2. The Cortical Learning Model

The "brain" is represented by a self-supervised neural network. Its objective is to predict the next ONS based on the current signal and inferred eye motion. By minimizing this prediction error, the model effectively "disentangles" the static hardware of the eye from the dynamic properties of the external scene.

Key Insights: Eye Motion as a Learning Signal

A central finding of this work is that fixational eye drift is not "noise" to be filtered out, but a critical feature for learning. Because the eye moves while the retinal mosaic remains fixed, the brain can use the resulting temporal fluctuations to infer the underlying structure of the input. This "disentanglement through ego-perturbation" allows the model to discover the correct color dimensionality (e.g., 3D for humans) without being explicitly programmed to do so.

Results: Emergent Dimensionality and Clinical Applications

The framework was tested using a dataset of 900 hyperspectral images of natural scenes (Arad et al., 2022). The results demonstrate that the model's internal representation naturally matches the dimensionality of the simulated retina.

Quantitative and qualitative analysis of emergent neural color, showing the model's ability to quantify color dimensionality (CMF-SIM) and visualize the resulting color space (NS).
  • Dimensionality Matching: If the simulated retina has $N$ cone types, the cortical model develops an $N$-dimensional internal color space. This was verified for monochromatic (1D), dichromatic (2D), trichromatic (3D), and even theoretical tetrachromatic (4D) systems.
  • Clinical Simulations: The model accurately reproduced color-space shifts characteristic of human color blindness (protanopia, deuteranopia, and tritanopia).
  • Vision Boosting: The model successfully simulated the results of gene therapy experiments in monkeys. By "re-learning" cortical parameters after adding a third cone type to a dichromatic retina, the model demonstrated an upgrade from 2D to 3D vision.

Conclusion and Future Work

This research provides a "complete existence proof" that low-level, self-supervised temporal prediction is sufficient to generate complex color vision. While the current model focuses on low-level emergence, future work aims to incorporate higher-level phenomena like color constancy and explore more biologically realistic learning rules beyond standard backpropagation. Furthermore, this logic of learning through ego-perturbation may be applicable to other sensory modalities, such as audition or somatosensation.

Summary updated

BibTeX
@inproceedings{kotani2025computational,
  author = {Atsunobu Kotani and Ren Ng},
  title = {{A Computational Framework for Modeling Emergence of Color Vision in the Human Brain}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=g3xuCtrG6H},
}
Figure