Traditionally, neural networks are viewed strictly as functions that transform input data into predictions. However, as thousands of pre-trained models and fine-tuned adapters are uploaded to public repositories daily, a new paradigm has emerged: treating the weights of these neural networks themselves as a rich, high-dimensional data modality.
This approach, known as weight-space learning, shifts the focus from using models to analyzing them directly through their parameters. By training meta-models to read and interpret raw weight tensors, researchers can uncover a model's training history, capabilities, and underlying structure without executing a single forward pass.
The Intuition: Weights as Cryptographic Footprints
Why should we learn from weights directly? As open-source model ecosystems expand, a significant challenge arises: many models lack comprehensive documentation regarding their training data, hyperparameters, or exact lineage. Dynamically evaluating these models on massive test suites to infer their properties is computationally prohibitive.
As argued by Eliahu Horwitz et al., a model's static weight tensors function as a physical footprint of its optimization journey. Every gradient step encodes information about the training data distribution, the loss landscape, and the optimization constraints directly into the parameters. Weight-space learning treats these parameters as features, allowing a meta-model to predict downstream properties—such as generalization error or training data origins—by recognizing the structural signatures left behind during training.
Extracting Features from Weight Spaces
Reading a neural network's "mind" from its raw parameters is notoriously difficult. Weights are highly unstructured, high-dimensional, and subject to permutation symmetries (e.g., swapping the order of hidden neurons in a layer doesn't change the network's function but radically alters the raw weight matrix). Simple linear projections often fail to capture the intricate, non-linear relationships embedded within these matrices.
To capture these complex correlations efficiently, Eunwoo Heo et al. developed a multi-view probing framework called MVProbe. Instead of flattening weight tensors into a single long vector, the authors combine traditional linear projections with second-order, Gram-based views.
$$\text{Gram Matrix} = W W^T$$
While a linear probe only evaluates individual weight magnitudes, the Gram matrix captures the covariance between different features or neurons within a layer. This second-order view provides a permutation-invariant signature of how features interact, making it significantly easier for a meta-model to recognize the subtle structural patterns that signify specific training conditions or data exposures.
Overcoming Data Scarcity in Weight Space
Training a meta-model to understand other neural networks requires a dataset where each individual data point is itself a fully trained model. This introduces a severe data scarcity problem: generating thousands of trained models to serve as training data is computationally exhausting, leading to severe overfitting in weight-space learners.
To mitigate this, Aviv Shamsian et al. introduced "Weight-Space Mixup." Mirroring traditional data augmentation in computer vision, Weight-Space Mixup linearly interpolates the weight tensors of two distinct trained models ($W_A$ and $W_B$) along with their respective target properties ($Y_A$ and $Y_B$):
$$W_{\text{new}} = \lambda W_A + (1 - \lambda) W_B$$ $$Y_{\text{new}} = \lambda Y_A + (1 - \lambda) Y_B$$
This geometric interpolation smooths the meta-learning landscape. By creating synthetic intermediate models, it prevents the meta-model from simply memorizing specific weight configurations, drastically reducing the number of explicitly trained models required to achieve high accuracy.
Benchmarking with Phase Transitions
To move weight-space learning beyond ad-hoc collections, researchers require structured, diverse populations of models. Addressing this need, Konstantin Schürholt et al. (and in their parallel dataset release Konstantin Schürholt et al.) introduced Phase Transition Model Zoos.
Rather than merely gathering final checkpoints, the authors systematically generated over 2,500 unique networks and 60,000 checkpoints across vision, language, and scientific computing. They mapped these models to distinct thermodynamic-like "phases" of their loss landscapes—such as underfitted, overfitted, or well-generalizing states. This benchmarking framework demonstrates that a model's physical location and trajectory in weight space fundamentally dictates its success in downstream optimization tasks like pruning, fine-tuning, and weight averaging.
[Raw Weight Tensors] ──► [Multi-View Probing (Linear + Gram)] ──► [Property Inference]
▲ │
│ (Augmented via Weight-Space Mixup) ▼
[Phase Transition Zoos] ─────────────────────────────────────────► [Model Atlas Mapping]
Macro Mapping and Micro Programming
Treating weights as a distinct data modality enables applications at both macroscopic and microscopic scales:
- The Macro Scale (Model Atlases): Eliahu Horwitz et al. propose charting a global "Model Atlas"—a comprehensive graph representation where every machine learning model is a node, and edges represent transformations like fine-tuning, pruning, or distillation. Weight-space learning serves as the foundational engine for this atlas, allowing automated systems to forensically discover relationships and trace the lineage of undocumented models.
- The Micro Scale (Algorithmic Compilation): Instead of using optimization to find weights, understanding the intrinsic language of parameter spaces allows for direct algorithmic design. Andy Yang et al. compiled the "Transformer Cookbook," formalizing mathematical recipes to manually program specific algorithms—such as formal logic, arithmetic, and data routing—directly into a Transformer's parameter matrices.
Historically, weight-based analysis has been fragmented across isolated sub-communities, ranging from specialized remote sensing tasks like differential weights-based band selection (Yun Liu 0014 et al.) to mechanistic interpretability. As highlighted by Konstantin Schürholt et al., unifying these disparate threads into a cohesive framework for weight-space learning is essential. By treating neural network parameters as a standardized data modality, the machine learning community can transition from treating models as opaque black boxes to analyzing them as structured, predictable, and searchable data assets.
Key Foundational Work
Data Augmentations in Deep Weight Spaces (2023) — contributed 1 concept elements to this direction.
Differential weights-based band selection for hyperspectral image classification (2017) — contributed 1 concept elements to this direction.