High-contrast "gaudy" images improve the training of deep neural network models of visual cortex

Benjamin Cowley, Jonathan W. Pillow

The Data Bottleneck in Computational Neuroscience

Deep Neural Networks (DNNs) currently serve as our most effective computational tools for modeling how the visual cortex—specifically areas like V4 and IT—transforms visual stimuli into neural signals. However, a fundamental bottleneck limits their application: DNNs are notoriously "data-hungry," often requiring thousands or millions of examples to learn generalized representations.

In stark contrast, neurophysiologists are severely constrained by experimental recording time. Collecting neural responses from a living brain is a slow, expensive process, with a typical experiment yielding responses to only a few thousand images. This creates a massive gap between the data required to train expressive, nonlinear models and the data actually available, frequently leading to overfitting. To bridge this gap, researchers must find ways to choose or synthesize training images that provide the maximum predictive value per recording second.

The "Gaudy" Solution

To address this challenge, researchers propose a simple, theory-driven manipulation to create "gaudy images"—high-contrast, binarized versions of natural images designed to serve as a highly efficient training set.

The generation process is straightforward:

  1. Binarization: Calculate the mean pixel intensity of a natural image.
  2. Extreme Contrast: Set every pixel intensity to either 0 or 255, depending on whether it falls below or above the mean.

The core intuition behind this approach is rooted in Active Learning (AL) theory. For linear models, the optimal training stimuli are "outliers" that maximize the variance of the input. Gaudy images act as synthesized outliers, pushing the model's hidden units to their limits and driving them into diverse response regimes that lower-contrast natural images rarely reach.

To test this, the authors utilized a transfer learning framework, employing a pre-trained DNN (like ResNet50) as a fixed feature extractor, and training a smaller "readout network" to map these features to neural responses.

Key Findings and Results

Performance of gaudy vs. normal images across different activation functions.

The efficacy of gaudy images was validated using both simulated "ground truth" responses from state-of-the-art DNNs and real electrophysiological recordings from macaque monkey area V4.

Driving Nonlinearity and Data Efficiency As illustrated in the figure above, gaudy images are particularly effective for training networks with strong nonlinearities. While they offer a modest improvement for linear models, their advantage becomes massive when training models with ReLU or Sigmoid activation functions. In simulations, training on a mix of gaudy and normal images reached higher accuracy levels much faster than training on normal images alone.

The Power of Edges Further analysis revealed that the effectiveness of gaudy images is primarily driven by high-contrast edges. The researchers found that transforming only the top 10% of pixels with the highest edge intensity achieves nearly the same training efficiency as a fully "gaudified" image.

Static Outperforms Dynamic Surprisingly, static gaudy images—chosen before training begins—outperformed several sophisticated active learning algorithms designed to dynamically choose images during training based on model uncertainty. This suggests that the features overemphasized by gaudy images are universally important for training visual models, negating the need for complex, adaptive selection algorithms.

Real-World Validation The benefits of this approach extend beyond simulations. When applied to real V4 neural data, models trained on naturally "gaudy-like" natural images (those with inherently high contrast and sharp edges) significantly outperformed models trained on a random selection of natural scenes. Furthermore, this advantage proved robust across multiple backbone architectures, including VGG and ResNet.

Limitations and Future Directions

While gaudy images offer a powerful tool for training predictive models, several open questions remain. Chief among them is biological realism: it is unclear whether real biological neurons might "saturate" or behave unpredictably when presented with such extreme stimuli, compared to the simulated units in a DNN.

Additionally, the current study focused on relatively small readout networks (approximately 1.4 million parameters). Future work must determine if this data-efficient strategy scales to training massive, end-to-end models from scratch. Finally, while briefly explored on datasets like CIFAR, the potential of gaudy images as a general data augmentation technique for standard computer vision tasks—or even their application to other domains like audio or medical signals—presents an exciting frontier for future research.

Summary updated

Abstract

A key challenge in understanding the sensory transformations of the visual system is to obtain a highly predictive model that maps natural images to neural responses. Deep neural networks (DNNs) provide a promising candidate for such a model. However, DNNs require orders of magnitude more training data than neuroscientists can collect because experimental recording time is severely limited. This motivates us to find images that train highly-predictive DNNs with as little training data as possible. We propose high-contrast, binarized versions of natural images---termed gaudy images---to efficiently train DNNs to predict higher-order visual cortical responses. In simulation experiments and analyses of real neural data, we find that training DNNs with gaudy images substantially reduces the number of training images needed to accurately predict responses to natural images. We also find that gaudy images, chosen before training, outperform images chosen during training by active learning algorithms. Thus, gaudy images overemphasize features of natural images that are the most important for efficiently training DNNs. We believe gaudy images will aid in the modeling of visual cortical neurons, potentially opening new scientific questions about visual processing.
BibTeX
@inproceedings{cowley2020high,
  author = {Benjamin Cowley and Jonathan W. Pillow},
  title = {{High-contrast "gaudy" images improve the training of deep neural network models of visual cortex}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=W6lkfuygNaU},
}
Figure