Towards Autoregressive Neural Network Parameter Generation

Bedionita Soro, Bruno Andreis, Song Chong, Sung Ju HwangarXiv 2026 · arXiv

TL;DR

What if we treated neural network weights like words in a sentence? This paper introduces Instruction-Guided Parameter Generation (IGPG), a framework that discretizes pretrained model weights into tokens and uses a Transformer to "write" new network parameters. By modeling weights autoregressively, IGPG can generate functional initializations for diverse architectures and unseen tasks, effectively transferring the "grammar" of successful models to new settings.

What Problem Does This Paper Address?

Adapting a neural network to a new task usually requires iterative, gradient-based optimization (fine-tuning), which is computationally expensive. While "hypernetworks" and generative weight-space models aim to synthesize parameters directly, they often fail when faced with:

  1. Heterogeneous Architectures: Most generators are hard-coded for a specific model structure.
  2. Diverse Task Distributions: Learning a single distribution that covers many different datasets is difficult.
  3. Out-of-Distribution (OOD) Generalization: Generating weights for a task or architecture the generator hasn't seen before remains a major hurdle.

The authors ask: Can we leverage the same autoregressive machinery that makes Large Language Models (LLMs) so good at text generation to generate neural network parameters?

Methods

The IGPG pipeline operates in two distinct stages, mirroring the "tokenize then model" approach of modern LLMs.

The IGPG pipeline: (A) Tokenization via VQ-VAE, (B) Modeling the weight-token language, and (C) Sampling parameters for new requests.
  1. Weight Tokenization (VQ-VAE): Continuous weights are flattened, chunked, and passed through a Vector Quantized Variational Autoencoder (VQ-VAE). This maps high-dimensional continuous parameters into a sequence of discrete "weight tokens" from a learned codebook.
  2. Instruction-Guided Modeling: A decoder-only Transformer (the "prior") is trained to predict the next weight token in a sequence. Crucially, this generation is conditioned on a soft prefix containing:
  3. Task Information: Dataset embeddings generated via CLIP.
  4. Architecture Information: Textual descriptions of the model (e.g., "ResNet with 3 blocks") encoded by an instruction-tuned LLM (Llama 3).
  5. Parameter Scaling: To handle models of varying sizes, weights are partitioned into fixed-size chunks. This allows a single generator to handle architectures spanning two orders of magnitude in parameter count.
Figure 2: Token format (one sequence per model). A continuous conditioning prefix dataset embedding eD and, when the zoo spans architectures, an architecture embedding eA is followed by a BOS token, the kl VQ-VAE codebook indices (grouped internally into l chunks/layers withk code each but not individually delimited), and a final EOS. The transformer is trained by next-token prediction with the prefix and padding excluded from the loss.

Datasets

The authors evaluated IGPG across several regimes:

  • Tiny Model Zoo: Small CNNs trained on MNIST, SVHN, CIFAR-10, and STL-10 for in-distribution benchmarking.
  • Meta-Album: A collection of 30 diverse datasets used to test transfer to unseen tasks.
  • ResNet & ViT Families: 125 ResNet variants and 160 Vision Transformer (ViT) configurations to test cross-architecture generalization.
  • LoRA Adapters: Checkpoints from LLMs (like Qwen) to demonstrate the framework's utility for parameter-efficient fine-tuning.

Key Insights and Ideas

  • Weights have "Grammar": The success of the autoregressive prior suggests that pretrained weights contain structural dependencies that can be learned similarly to natural language.
  • The Prior is the Bottleneck: By disentangling reconstruction error (from the VQ-VAE) and generation error (from the Transformer), the authors found that for OOD architectures, the Transformer's inability to predict the correct "weight-token grammar" is a much bigger issue than the VQ-VAE's ability to represent the weights.
  • Initialization over Zero-Shot: While IGPG doesn't yet produce "perfect" models for unseen tasks out of the box, it provides initializations that are significantly closer to the solution than random initialization, leading to faster and better convergence during fine-tuning.

Key Results

  • In-Distribution Excellence: On the Tiny Model Zoo, IGPG achieved performance competitive with (and often exceeding) state-of-the-art diffusion-based and KDE-based weight generators.
  • Superior Initializations: For unseen datasets like Oxford Pets and FGVC Aircraft, IGPG initializations reached ~46% accuracy after just one epoch of fine-tuning, whereas random initialization stayed below 23%.
Figure 3: Accuracy on unseen datasets before fine-tuning and after one fine-tuning epoch. Error bars show std over IGPG runs.
  • Architectural Scaling: The framework successfully generated initializations for a ViT-B/16 (85.2M parameters), despite being trained on much smaller models (under 9M parameters). This initialization improved final Top-1 accuracy on CIFAR-10 by 0.59% over random starts.
  • LoRA Generation: IGPG was able to generate effective LoRA adapters for ViT-Base, outperforming specialized methods like FourierFT by 5.7% on average across six vision benchmarks.

Limitations and Future Work

The authors acknowledge that we aren't at "GPT for Weights" yet. Current limitations include:

  • Scale: The experiments were limited to models with tens of millions of parameters. Scaling to billions of parameters will require more efficient tokenization and massive "weight-token" corpora.
  • Zero-Shot Gap: For truly unseen tasks, the model still requires some downstream adaptation.
  • Prior Generalization: Improving the Transformer's ability to generalize to entirely new architectural families (e.g., moving from CNNs to Transformers) remains a challenge.

Future work points toward stronger prior pretraining and exploring alternative tokenization schemes to better capture the functional properties of neural networks.

Summary updated

Abstract

Large Language Models (LLMs) have recently established a new paradigm in machine learning, demonstrating exceptional zero-shot generalization and complex reasoning capabilities. Concurrently, neural network parameter generation has emerged as a promising avenue for rapid task adaptation and model synthesis. Existing weight generation techniques, however, struggle to jointly learn the distribution of diverse models, to encode heterogeneous architectures within a single generator, and to generalize out of distribution. We therefore ask whether pretrained parameters can be encoded into a discrete domain so that the zero-shot generalization LLMs exhibit in text generation can be transferred to weight generation. Following this view, we propose an LLM-inspired approach, Instruction-Guided Parameter Generation (IGPG), that formulates network parameter generation as an autoregressive modeling problem. We first quantize pretrained weights with a VQ-VAE, then train a prefix-conditioned autoregressive generator over the resulting tokens. At inference, the generated weight tokens are mapped back to parameter space by the VQ-VAE decoder. We evaluate the framework across in-distribution generation, cross-dataset transfer, and cross-architecture settings. The proposed method achieves competitive performance in in-distribution generation and provides useful parameter initializations for held-out datasets and architectural configurations, improving downstream performance over random initialization after adaptation. These results suggest that discrete autoregressive modeling provides a promising framework for learning and transferring structure across collections of pretrained neural network parameters.
BibTeX
@misc{soro2026towards,
  author = {Bedionita Soro and Bruno Andreis and Song Chong and Sung Ju Hwang},
  title = {{Towards Autoregressive Neural Network Parameter Generation}},
  year = {2026},
  eprint = {2611.25538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25538},
}
Figure