Traditionally, neural networks are treated as functions that process data, while their internal weights are viewed as unstructured, high-dimensional vectors optimized via gradient descent. However, a growing body of research treats neural network weights themselves as a distinct data modality—a field known as Weight Space Learning (WSL).
The central challenge of WSL is scalability and structure. A modern neural network contains millions or billions of parameters arranged in a strict hierarchical architecture of layers, channels, and blocks. Flattening these parameters into a single massive vector destroys this geometric structure and creates an intractable input size for standard machine learning models.
To overcome this, researchers have introduced parameter tokenization. By partitioning neural network weights into structured sequences of tokens, standard sequence modeling architectures—such as Transformers and Diffusion Models—can analyze, interpret, and generate functional neural network weights.
The Mechanics of Parameter Tokenization
To process a network's weights as a sequence, the continuous parameters must be broken down into discrete, manageable chunks while preserving their architectural context. This is typically achieved through a multi-step pipeline:
- Partitioning (Chunking): Instead of flattening the entire network, weights are divided by their structural boundaries. Schürholt in Towards Scalable and Versatile Weight Space Learning demonstrates that partitioning parameters into discrete subsets (such as layers or channels) allows weight-space models to scale to large architectures like ResNet-101 by avoiding the computational bottlenecks of global embeddings. Alternatively, Amaduzzi et al. process individual rows of weight matrices to extract low-level features from pre-trained networks.
- Handling Dimensional Asymmetry: Neural network layers vary widely in size. To feed them into a fixed-size sequence model, Andreis et al. propose a "pad-chunk-encode" pipeline. This approach pads varying layer dimensions to a uniform size and splits them into smaller parallel chunks, allowing a fixed-size encoder to ingest diverse architectures without losing structural information.
- Injecting Structural Geometry: Once weights are tokenized, their original position within the network hierarchy must be preserved. Gong et al. utilize sinusoidal positional encodings to help generative models distinguish between different layers and functional components. To capture both depth and width, Wang et al. introduce two-dimensional positional embeddings that explicitly represent layer and token indices, ensuring the model maintains spatial and hierarchical context across large-scale parameter sets.
Why Sequence Modeling Works in Weight Space
The core reason parameter tokenization succeeds is that it aligns the structural properties of neural networks with the inductive biases of self-attention mechanism.
Neural networks exhibit complex internal dependencies and permutation symmetries (e.g., swapping two neurons in an intermediate layer and adjusting the adjacent weights leaves the network's function unchanged). As Schürholt et al. show in Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction, treating neuron-specific weight groups as tokens allows self-attention to map these structural dependencies across disparate layers.
Furthermore, processing weights layer-wise prevents the catastrophic information loss associated with global flattening. Cao et al. observe that organizing weights and biases into layer-wise tokens significantly improves the convergence and output quality of generative models, because the Transformer can learn the statistical distributions of parameters relative to their specific architectural roles.
Applications: From Analysis to Generation
Once weights are represented as sequences of tokens, the entire toolkit of modern sequence modeling can be applied to weight spaces, enabling both discriminative and generative tasks.
1. Predicting Model Characteristics
By encoding weights into tokens, meta-models can "read" a network's parameters to predict its performance without running inference on a validation dataset. Falk et al. leverage this representation to perform zero-shot knowledge transfer and predict generalization gaps across heterogeneous model distributions. In multimodal settings, Amaduzzi et al. feed tokenized Neural Radiance Field (NeRF) weights directly into a language model, allowing the system to reason about 3D spatial relationships without ever rendering a 2D image.
2. Conditional Weight Generation (Text-to-Model)
Instead of optimizing weights via gradient descent, models can generate functional parameters directly from instructions. Li et al. and Tian et al. utilize diffusion transformers operating on partitioned weight blocks to synthesize custom, high-quality classifiers directly from natural language prompts. Similarly, Soro et al. combine a VQ-VAE with an autoregressive Transformer to discretize and "hallucinate" functional weights based on textual task descriptions.
[Text Prompt / Instruction]
│
▼
[Transformer / Diffusion Model] ──► Processes Layer-wise Tokens ──► [Functional Target Weights]
▲
│
[2D Positional Encodings] (Layer & Token Indices)
3. Accelerating Optimization and Personalization
Tokenized weight generation can bypass or drastically accelerate traditional training loops. Gong et al. use a diffusion model over weight blocks to predict ideal initialization parameters, achieving a 15x training speedup for image-to-image translation tasks. For model personalization, Ruiz et al. model neural network layers as a sequence within a Transformer decoder to predict specialized weights for text-to-image models, reducing personalization time to a matter of seconds.
4. In-Context Execution
Beyond analyzing and generating static weights, tokenization allows frozen models to dynamically alter their behavior. Hu et al. demonstrate that encoding the parameters of a target network directly into a sequence of prompt tokens allows a fixed-weight Transformer to retrieve and execute those parameters during inference, effectively treating the prompt tokens as an executable program.
Key Foundational Work
HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion (2023) — contributed 2 concept elements to this direction.
Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021) — contributed 1 concept elements to this direction.