TL;DR
This paper proposes and tests the hypothesis that sensory cortex neurons are optimized to predict future sensory input. By training artificial neural networks to predict future visual and auditory stimuli from past inputs, the researchers found that the resulting "receptive fields" closely matched those of real neurons, particularly in their temporal characteristics. This suggests that the brain's sensory processing is fundamentally geared towards anticipating what's coming next.
What Problem Does This Paper Address?
The fundamental question this paper addresses is: why do neurons in sensory cortices (like the visual and auditory cortex) develop the specific tuning properties they do? For instance, why are visual neurons selective for oriented edges moving at certain speeds, and auditory neurons for specific frequency changes? While principles like "sparse coding" (efficiently representing information with minimal neural activity) have explained some aspects, they haven't fully accounted for the temporal dynamics observed in real neurons. This paper explores an alternative or complementary hypothesis: that sensory processing is optimized for prediction of future sensory input, a crucial ability for guiding actions in a world with inherent processing delays.
Methods
The core of the study involves training simple feedforward artificial neural networks. These networks have a single hidden layer, mimicking a simplified cortical processing stage. The key innovation is the training objective: instead of optimizing for efficient representation of the past input (like sparse coding), the networks are trained to predict the immediate future of the sensory input (video frames or audio spectrograms) based on a sequence of past inputs.
- Architecture: A standard feedforward neural network with a linear-nonlinear (LN) encoder and a linear decoder.
- Training Objective: Minimizing the mean squared error between the predicted future and the actual future sensory input.
- Regularization: L1 weight regularization is used to encourage a parsimonious network, driving redundant weights towards zero. This is applied to the weights, not the activations, and differs from typical sparse coding.
- Comparison Models: A sparse coding model is trained as a control to compare against the temporal prediction model.
Datasets
- Visual Inputs: Wildlife videos were used. These were processed into 20x20 pixel patches, band-pass filtered, downsampled, and cut into clips of 8 frames (320 ms) each. The network was trained to predict the last frame (future) from the first seven frames (past). Approximately 500,000 training clips and 100,000 validation clips were used.
- Auditory Inputs: Natural sounds, including human speech, animal vocalizations, and environmental sounds, were compiled. These were converted into "cochleagrams" (approximating auditory nerve activity) and processed into sequential clips of 43 time steps (415 ms). The network was trained to predict the last three time steps (15 ms) from the first 40 time steps (200 ms). Approximately 1,000,000 training clips and 200,000 validation clips were used.
- Real Neural Data: Spectrotemporal receptive fields (RFs) from auditory neurons (A1 and AAF) in ferrets and spatiotemporal RFs from V1 simple cells in cats, mice, and monkeys were used for comparison.
Key Insights and Ideas
- Prediction as a Unifying Principle: The study strongly suggests that the temporal tuning properties of sensory neurons, particularly their asymmetry and preference for recent inputs, are a direct consequence of the brain's need to predict future sensory information.
- Temporal Asymmetry is Key: Unlike previous models that often produced symmetric temporal receptive fields, the temporal prediction model naturally generated asymmetric RFs, mirroring real neurons' greater sensitivity to recent past events.
- Model RFs Mimic Real Neurons: The artificial networks, when trained to predict the future, developed receptive fields that closely resembled those of real neurons in both visual and auditory cortices, capturing spatial, frequency, and crucially, temporal tuning.
- Prediction Accuracy Correlates with Realism: The better the network was at predicting future input, the more its generated receptive fields resembled those of actual neurons. This provides a strong, independent criterion for model optimization.
- Beyond Sparse Coding: While sparse coding is a successful principle, temporal prediction offers a complementary objective that better explains the temporal dynamics observed in sensory cortices.
Key Results
- Auditory Cortex: The temporal prediction model produced spectrotemporal RFs that were qualitatively and quantitatively similar to real A1 neurons, including asymmetric temporal envelopes with lagging inhibition.
- Visual Cortex: The model generated spatiotemporal RFs that closely matched V1 simple cells, capturing both space-time separable and inseparable structures, as well as temporal asymmetry.
- Quantitative Comparison: Multi-dimensional scaling (MDS) showed that the RFs of the temporal prediction model and real neurons occupied similar regions in a feature space, while sparse coding model RFs spanned a much larger, distinct region.
- Hyperparameter Optimization: The study found a strong correlation (r² = 0.8) between the model's prediction error and the similarity of its RFs to real neuronal RFs, supporting the prediction hypothesis. Optimal prediction settings yielded the most biologically realistic RFs.
Limitations and Future Work
- Model Simplification: The model uses a simple feedforward network with a single hidden layer. More complex architectures (e.g., with recurrent connections or multiple layers) could potentially account for more complex tuning properties and higher-level sensory processing.
- Dataset Specificity: Differences between model and real neuron RFs (e.g., narrower frequency tuning in auditory models, overrepresentation of vertical/horizontal orientations in visual models) might stem from the specific datasets used for training. Future work could explore the influence of datasets with different statistics and match them more closely to the evolutionary context and individual experience of the animals.
- Recording Location and Sampling Bias: Differences could also arise from the specific cortical layers sampled and potential biases in neuron sampling methods (e.g., multi-electrode arrays favoring larger neurons).
- Hierarchical Prediction: The authors suggest developing hierarchical versions of the temporal prediction model to explain neuronal responses in higher brain areas.
- Applications: The learned features could potentially accelerate learning in tasks like speech or object recognition, by providing initialization for supervised or reinforcement learning networks.
- Biological Plausibility: While the model uses backpropagation, which has biological plausibility questions, the authors note progress in developing trainable networks with more biologically plausible characteristics. They also speculate on potential biological analogues for the model's components (input units as thalamic input, hidden units as cortical neurons, output units representing prediction or prediction error).
Summary updated