Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules

Kazuki Irie, Jürgen Schmidhuber

Introduction

This research introduces Fast Weight Painters (FPAs), a novel generative model that conceptualizes images as weight matrices within a "fictive" neural network. By leveraging sequential synaptic learning rules—specifically the delta rule—to construct these matrices one rank at a time, the authors demonstrate that generic learning mechanisms can synthesize complex, human-interpretable images without the explicit inductive biases of standard convolutional networks.

The Challenge of Interpretability in Weight Generation

In the domain of "Fast Weight Programmers" (FWPs), a "slow" neural network generates or modifies the weights of a "fast" network to store information or execute specific programs. While effective for tasks like language modeling and reinforcement learning, the resulting weight matrices are typically "black boxes." Humans cannot easily decipher the knowledge or patterns encoded within these numerical arrays.

The authors propose a shift in perspective: What if we use these weight-generation principles to create visual outputs? By tasking a network to "program" a weight matrix that corresponds to a natural image, we can visually observe how abstract learning rules construct complex patterns step-by-step. This approach bridges the gap between neural architecture theory and interpretable computer vision.

Methodology: The Fast Weight Painter (FPA)

The core innovation is the FPA, which "paints" an image over $T$ discrete steps rather than generating it in a single forward pass.

Images as Weights

A color image is represented as three distinct weight matrices, one for each RGB channel. The generation process treats the image as a generic high-dimensional matrix.

The Delta Rule for Sequential Updates

Unlike simple additive (Hebbian) rules, the FPA employs the delta learning rule. At each step $t$, the model generates a "key" ($k_t$), a "value" ($v_t$), and a learning rate ($\beta_t$). The update is defined as: $$W_t = W_{t-1} + \sigma(\beta_t)(v_t - W_{t-1}\phi(k_t)) \otimes \phi(k_t)$$ This formulation allows the model to inspect its current progress ($W_{t-1}$) and apply corrective updates, effectively refining the image iteratively.

Architecture of the Fast Weight Painter (FPA) showing the sequential update process using the delta rule.

Architecture and Refinement

The system utilizes a latent vector processed by an LSTM-based "Sequence Processor" to output instructions for each painting step. To further enhance image quality, the authors incorporate a convolutional U-Net at the end of the sequence to perform "one-step denoising." The entire framework is trained using a Generative Adversarial Network (GAN) objective.

The FPA pipeline followed by a U-Net refinement step to produce high-quality final images.

Key Insights

  • The Power of Correction: The delta rule's subtraction term ($v_t - W_{t-1}\phi(k_t)$) is crucial. It enables the model to "erase" or modify previous strokes, leading to significantly higher image quality compared to purely additive methods.
  • Visualizing the "Mind's Eye": The sequential nature of the FPA allows for real-time visualization of the generation process. The model often begins with a global "sketch" before adding fine details like hair or eyes, or sometimes adopts a printer-like column-by-column approach.
  • Symmetry Exploitation: In low-rank settings (fewer steps), the model learns to exploit natural symmetries, such as painting both eyes of a face simultaneously in a single update step.

Experimental Results

The FPA was evaluated on several 64x64 benchmarks, including CelebA, LSUN-Church, and AFHQ.

  • Image Quality: On CelebA, the base FPA achieved an FID score of 18.3.
  • U-Net Synergy: The addition of the U-Net refinement step dramatically improved performance, reducing the CelebA FID to 3.7, comparable to specialized lightweight GAN architectures.
  • Step Dependency: Quality scales with the number of steps ($T$). Reducing $T$ from 64 to 8 significantly degrades the FID, though images remain recognizable as low-rank approximations.

Conclusion and Future Work

The FPA serves as a proof of concept that generic synaptic learning rules are powerful mechanisms for pattern generation. While it does not yet outperform highly optimized convolutional architectures in raw quality, it offers a unique window into the internal logic of weight-generating networks. Future research may explore scaling this process to higher resolutions and investigating the mathematical parallels between rank-1 updates and diffusion-based denoising.

Summary updated

BibTeX
@inproceedings{irie2023images,
  author = {Kazuki Irie and Jürgen Schmidhuber},
  title = {{Images as Weight Matrices: Sequential Image Generation Through Synaptic Learning Rules}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ddad0PNUvV},
}
Figure