TL;DR
This paper introduces a novel dynamic predictive coding model that uses hypernetworks to learn time-varying transition dynamics for spatiotemporal prediction of natural video sequences. This approach allows the model to learn hierarchical representations with different timescales, mimicking observed neural processing in the visual cortex and generating V1-like receptive fields.
What Problem Does This Paper Address?
Traditional predictive coding models, like Rao and Ballard's seminal work, primarily focused on spatial prediction to explain receptive field properties. However, they struggled to capture the temporal dynamics of natural scenes. Existing models for spatiotemporal prediction often relied on static transition matrices or were not hierarchical, limiting their ability to model the complex, time-varying nature of visual input. The paper aims to bridge this gap by developing a hierarchical model that can learn and predict these dynamic temporal relationships, addressing the question of how the brain might achieve spatiotemporal prediction and exhibit hierarchical timescales.
Methods
The core innovation is the use of hypernetworks to dynamically modulate transition matrices. Instead of a fixed transition matrix, a hypernetwork learns to generate time-varying weights that mix a set of learned "basis" transition matrices. This allows the model to adapt its predictions to changing input dynamics.
The authors propose two main model architectures:
- Single-Level Recurrent Model: A single layer of predictive coding neurons where a recurrent hypernetwork modulates the transition matrix based on the current and past hidden states.
- Two-Level Hierarchical Model: A hierarchical structure where a higher-level (second) layer generates the transition dynamics for the lower (first) layer via a feedforward hypernetwork. This allows the higher level to learn to predict the dynamics of the lower level, rather than just its state.
Both models are trained by minimizing spatiotemporal prediction errors, effectively performing Bayesian inference and learning through gradient descent.
Datasets
The models were trained and evaluated on:
- Natural Video Segments: 5,000 segments from a YouTube video of a forest trail walk. These were spatially and temporally whitened.
- Moving MNIST Dataset: Sequences of handwritten digits moving within a frame, used to illustrate how prediction errors drive changes in the higher level.
Key Insights and Ideas
- Dynamic Transitions are Crucial: The ability to dynamically adjust transition matrices is key to accurately predicting complex natural video sequences, outperforming static models, especially when input corrections are removed.
- Hypernetworks as Dynamic Modulators: Hypernetworks provide a neurally plausible mechanism for context-dependent modulation of synaptic weights, allowing for flexible adaptation to changing environmental dynamics.
- Emergent Hierarchical Timescales: The hierarchical model naturally learns representations with different timescales. Lower levels respond quickly to input changes, while higher levels exhibit greater stability and capture longer-term dynamics, mirroring observations in the visual cortex.
- Generative Power of Higher Levels: The higher-level representation in the hierarchical model can generate diverse sequences of lower-level dynamics, suggesting it learns a rich internal model of how visual scenes evolve.
Key Results
- Improved Prediction Accuracy: The dynamic model significantly outperformed static transition matrix models in mean squared error for next time-step image prediction on both training and test sets.
- V1-like Receptive Fields: The first-level neurons in both the single-level and hierarchical models learned space-time receptive fields (STRFs) that closely resemble those of simple cells in the primary visual cortex (V1), including both separable and inseparable (direction-selective) types.
- Hierarchical Stability: In the hierarchical model, the second-level neurons showed remarkable stability and longer timescale responses compared to the first-level neurons, especially when the input dynamics were predictable. This stability emerged naturally from minimizing prediction errors.
- Generative Capabilities Demonstrated: Sampling different higher-level states in the hierarchical model led to the generation of diverse video sequences with varying forms, motion, and speeds, showcasing the model's learned generative capabilities.
Limitations and Future Work
- Model Complexity: The use of hypernetworks adds complexity to the model.
- Future Work:
- More quantitative comparisons of learned STRFs with those found in the visual cortex.
- Exploring deeper hierarchical models to explain response properties of higher-order cortical areas.
- Further investigation into the role of hypernetworks as a mechanism for neuromodulation of cortical synapses.
Summary updated