Scaling Generative Models to Neural Network Weight Spaces

125 concepts102 papers1 proposal

Standard deep learning treats every model training run as an isolated optimization problem. To learn a new task, we initialize a network with random noise or a generic pre-trained checkpoint, and then slowly guide its parameters across a loss landscape using Stochastic Gradient Descent (SGD).

But what if we zoomed out and looked at the space of all possible trained models? In this paradigm, a fully trained neural network is simply a single data point living on a massive, high-dimensional manifold of functional models. If we can learn the underlying distribution of this manifold, we can use generative models to sample entirely new, high-performing neural network parameters instantly, bypassing SGD altogether.

This idea powers a growing body of research into weight-space learning and parameter generation. However, moving from generating pixels to generating parameters introduces a fundamental hurdle: scalability.

The Scalability Bottleneck: Dimensions and Symmetries

Why is generating neural network weights so difficult compared to generating images or text? The challenge stems from two core properties of weight spaces:

  1. Extreme Dimensionality: Even a lightweight convolutional network contains millions of parameters, while modern transformers span billions. Flattening these weights into a single vector creates a target space that causes standard generative architectures to fail due to memory and computational bottlenecks.
  2. Permutation Symmetries: Neural networks possess massive structural redundancies. In any hidden layer, you can swap the positions of two neurons (and reorder their corresponding incoming and outgoing weights) without changing the network's actual mathematical function. This means that completely different weight matrices can represent identical behavior, making the weight-space manifold highly disjoint, non-convex, and difficult for standard generative models to map.

To overcome these challenges and scale parameter generation to practical architectures, researchers have developed three core strategies: sequential tokenization, latent subspace generation, and behavioral alignment.

Breaking the Dimensionality Barrier via Tokenization

Instead of treating an entire neural network as one giant, unwieldy vector, modern scalable approaches decompose the weight space into structured chunks or tokens, borrowing architectures directly from natural language processing.

[Raw Weight Matrices] ──► [Decompose into Blocks] ──► [VQ-VAE Tokenization] ──► [Autoregressive Transformer]

In Towards Scalable and Versatile Weight Space Learning, Konstantin Schürholt addresses the global embedding bottleneck by proposing SANE, a framework that treats neural network weights as a sequence of localized tokens. By processing weights sequentially rather than compressing the entire parameter vector into a single global representation, this approach successfully scales weight-space learning to larger architectures like ResNet-101.

Building on this sequential intuition, Bedionita Soro et al. introduce the IGPG framework in Instruction-Guided Autoregressive Neural Network Parameter Generation. The authors utilize a Vector Quantized Variational Autoencoder (VQ-VAE) to discretize neural weight blocks into a sequence of discrete tokens. Once the continuous weight space is mapped to a discrete vocabulary, an autoregressive Transformer can "hallucinate" valid neural network parameters based on textual task instructions and architectural specifications.

Diffusion and Flow Matching on the Weight Manifold

Once a weight space is tokenized or compressed, the next challenge is choosing the right generative mechanism to sample from it. Diffusion models and flow matching have proven uniquely capable of navigating these complex manifolds.

In Neural Network Diffusion, Boya Zeng et al. demonstrate that parameter synthesis can be framed as an iterative denoising process. Their framework takes random noise matrices and progressively refines them into functional parameters that achieve high performance on target datasets.

To scale this concept to massive architectures where full-parameter generation remains computationally prohibitive, Kai Wang et al. propose COND P-DIFF in Conditional LoRA Parameter Generation. Instead of generating the entire backbone of a model, the authors restrict the diffusion process to synthesizing Low-Rank Adapters (LoRA). This drastically reduces the target dimensionality, making the generation of parameters for large-scale models hardware-feasible.

However, the problem of permutation symmetry still looms large: if two identical models in a training zoo have permuted neurons, a diffusion model attempting to interpolate between them will average their weights, resulting in a broken, non-functional model. To resolve this, Saumya Gupta et al. introduce DeepWeightFlow in DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weights. The authors utilize a "re-basining" mechanism that mathematically aligns the neuron permutations of different models before training a flow-matching generator. By mapping all source models to a shared canonical basis, the generator can smoothly model the parameter trajectories without destructive interference from weight symmetries.

Structure vs. Behavior: The Functional Constraint

A final, crucial intuition in scaling parameter generation is recognizing that structure does not equal behavior. In traditional generative modeling (like images), a small pixel error results in a slightly blurry image. In weight space, a minor numerical shift in a single weight matrix can completely alter the network's internal activations, rendering the generated model useless.

Raw Weight Loss (MSE) ──► Matches numbers, but ignores network logic ──► High failure rate
Behavioral Loss       ──► Matches model outputs on actual data      ──► Functional preservation

This limitation is explored by Léo Meynent et al. in Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstruction. The authors show that autoencoders trained solely on a structural loss (such as mean squared error on the raw weight values) fail to preserve the performance of the reconstructed models. By introducing a behavioral loss—which penalizes the difference between the outputs of the original network and the reconstructed network on actual data—the framework ensures that the synthesized parameters maintain their functional integrity.

This focus on function over raw structure aligns with earlier insights from NetSpace, a framework introduced by Gianluca Berardi et al. in Learning the Space of Deep Models. NetSpace embeds trained models into a low-dimensional latent space optimized specifically for functional similarity. By prioritizing how a model behaves over how its raw weights look, it becomes possible to navigate, interpolate, and sample within the model manifold reliably.

By combining sequential tokenization to handle dimensionality, flow matching with re-basining to handle symmetries, and behavioral losses to preserve function, modern meta-models are transforming neural network weights from abstract mathematical configurations into a structured, searchable, and instantly synthesizable modality.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 106 papers
  2. Version 1 · 2026-10-04 · 200 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_generation_scalability_f18d5eec,
  title = {parameter generation scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-generation-scalability/rdr_f18d5eeca08d4f34b0a9b54fc34c6948},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_parameter_generation_scalability_f18d5eec_v2,
  title = {parameter generation scalability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-generation-scalability/rdr_f18d5eeca08d4f34b0a9b54fc34c6948/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake