Abstract
This paper investigates the generative dynamics of Restricted Boltzmann Machines (RBMs) as a proxy for spontaneous brain activity. By initiating the generation process from "chimera states"—hidden representations that blend features of multiple digit categories—the authors demonstrate that energy-based models can be nudged to explore a more diverse range of internal states. The study highlights the inherent tension between attractor stability and metastability in neural networks, revealing that while specific initialization strategies can enhance state exploration, these models are prone to decaying into "non-digit" noise sinks.
Introduction
In neuroscience, "spontaneous brain activity" refers to neural firing that occurs when the brain is at rest. A leading hypothesis suggests this activity represents the brain's internal generative model sampling from its learned priors. However, standard energy-based models like Restricted Boltzmann Machines (RBMs) struggle to replicate the fluid, heterogeneous nature of this activity. Because they are governed by an energy landscape, they tend to fall into "attractor states" (local energy minima) and remain there. For instance, once an RBM starts generating a specific digit, it is mathematically difficult for it to transition into another without external input. This research investigates whether specific initialization strategies can increase the "metastability" of these networks, allowing them to visit more states in a single generative run.
Methods
The researchers utilized a standard RBM architecture (784 visible units, 1000 hidden units) trained on the MNIST dataset. To analyze generative dynamics, they employed several key techniques:
- Top-down Sampling: Instead of starting with an image, the process begins with a hidden state, allowing the model to iteratively generate visible and hidden patterns.
- Label Biasing: A linear classifier inversion technique was used to identify the specific hidden state corresponding to a "pure" digit.
- Chimera States: To encourage exploration, the authors developed two methods to create "biased" hidden states that sit between two digit categories: the Intersection Method (activating shared hidden units) and Double Label Biasing (inverting a classifier with two active digit labels).
- The "Observer" Classifier: A separate VGG-16 convolutional neural network was trained to classify generated images into 11 categories (digits 0-9 plus a "non-digit" noise class) to objectively measure the model's internal state.
Key Results
The study found that standard RBMs are remarkably "sticky." When started from a standard digit bias, the model visited an average of only 1.779 distinct digit states per 100-step generation period. However, starting the model in a "chimera state" via the Intersection Method significantly increased this exploration to 2.951 states on average. This suggests that the boundaries between attractors in the energy landscape are where the most dynamic transitions occur.
Quantitative analysis revealed that the probability of staying within the same digit state was high (~0.87), while transitions to different digits were rare (~0.01). Furthermore, the model exhibited significant "drift"; classifier accuracy for generated images dropped to approximately 11.2% by step 100 (with the exception of the digit "0"). A critical finding was the "Non-Digit Sink": once the model entered a noisy, unrecognizable state, it almost never returned to a recognizable digit, indicating that noise is a powerful attractor in the RBM's landscape.
Discussion and Future Work
While chimera states improve exploration, the RBM's dynamics remain far less flexible than a biological brain, which can transition through a vast variety of states without decaying into pure noise. The model did, however, maintain a consistent level of sparsity (14–22% active hidden units), mirroring biological sparse coding.
Future research directions include:
- Gaussian-Bernoulli RBMs: Utilizing models that handle continuous input to produce more realistic, less noisy transitions.
- Complex Datasets: Moving beyond MNIST to natural images to explore richer attractor landscapes.
- Temporal Constraints: Implementing architectures that explicitly model the timing of transitions to better match electrophysiological data from biological systems.
Summary updated