A Survey of Weight Space Learning: Understanding, Representation, and Generation

Xiaolong Han, Zehong Wang, Bo Zhao, Binchi Zhang, Jundong Li, Damian Borth, Rose Yu, Haggai Maron, Yanfang Ye, Lu Yin, Ferrante Neri

Abstract

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. This emerging research direction, Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL, categorizing existing methods into Weight Space Understanding (WSU), Weight Space Representation (WSR), and Weight Space Generation (WSG).

Introduction

In the traditional deep learning paradigm, weights are considered the static output of an optimization process. While research has historically prioritized input data and output predictions, the weights themselves are often treated as a black box. However, the proliferation of "model zoos"—repositories such as Hugging Face containing millions of pretrained models—has introduced a new challenge: the systematic organization, comparison, and reuse of this vast collective knowledge. Weight space is not a random collection of parameters; it possesses a deep mathematical structure governed by symmetries and optimization dynamics. This survey consolidates fragmented research into the unified discipline of Weight Space Learning (WSL).

Core Dimensions of Weight Space Learning

Weight Space Understanding (WSU)

WSU focuses on the underlying "physics" of weight space, specifically studying Invariance (different weight configurations producing identical functions) and Equivariance (predictable output changes resulting from weight transformations). Key techniques involve the use of quotient spaces to eliminate redundancies caused by symmetries, such as neuron permutations.

Weight Space Representation (WSR)

WSR involves learning latent embeddings of models for downstream analysis.

  • Model-based approaches: These utilize specialized architectures like Graph Neural Networks (GNNs) or CNNs to process weight tensors directly as structured data.
  • Model-free approaches (probing): These bypass raw weights to create a "behavioral signature" by observing model responses to specific input sets.
Model-based approaches process weight tensors directly to learn embeddings.

Weight Space Generation (WSG)

WSG moves from analysis to the synthesis of new models.

  • Hypernetworks: Small auxiliary networks that output weights for a larger "target" network in a single forward pass.
  • Generative Models: The application of Diffusion models or GANs to sample new, functional weight configurations from a learned distribution of pretrained models.
Weight Space Generation enables the synthesis of entire models through generative processes.

Datasets and Benchmarks

The primary data source for WSL is the Model Zoo. Key benchmarks include:

  • MLP/CNN Zoos: Collections of thousands of small networks trained on standard datasets like MNIST or CIFAR-10.
  • Implicit-Zoo: Focuses on Implicit Neural Representations (INRs) where weights represent signals such as images or 3D shapes.
  • LoRA Zoos: Modern benchmarks storing Low-Rank Adaptation weights for large-scale models like Stable Diffusion or Large Language Models (LLMs).

Key Insights: Weights as a First-Class Modality

A central insight of WSL is that weights are "learners of data," encoding architectural inductive biases and optimization trajectories. A significant challenge is the Permutation Problem, where swapping neurons in a hidden layer preserves functionality but alters weight values. WSU is critical for "aligning" models to enable fair comparison. Furthermore, architecture-agnostic probing allows for the comparison of functionally similar models, such as CNNs and Transformers, within a shared latent space.

Practical Applications and Results

WSL enables several novel capabilities:

  • Model Retrieval: Identifying the optimal pretrained model for a new task via embedding space search.
  • Zero-Shot Prediction: Meta-models predicting test accuracy by analyzing weights without access to test data.
  • Training Acceleration: Utilizing weight generators (e.g., GHN-3) to produce near-optimal initial weights, significantly reducing training time.
  • Data-Free Reconstruction: Reconstructing training data from model weights, which has significant implications for privacy research.

Limitations and Future Directions

Current WSL methods face a Scaling Wall, as processing the billions of weights in modern LLMs is computationally prohibitive. Modular processing of layers or LoRA adapters is a proposed path forward. Additionally, real-world features like Batch Normalization break simple mathematical symmetries, complicating theoretical analysis. Future research aims to develop "Universal Weight Processors" and address the adversarial risks associated with generated models.

Summary updated

Abstract

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.
BibTeX
@misc{han2026survey,
  author = {Xiaolong Han and Zehong Wang and Bo Zhao and Binchi Zhang and Jundong Li and Damian Borth and Rose Yu and Haggai Maron and Yanfang Ye and Lu Yin and Ferrante Neri},
  title = {{A Survey of Weight Space Learning: Understanding, Representation, and Generation}},
  year = {2026},
  eprint = {2603.10090},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.10090},
}
Figure