Learning Latent Representations of Neural Network Weights

This direction is split into other directions: latent parameter compression. Its last version stays readable and citable here.

Version 1 of 157 papers61 conceptsfrozen 2026-10-04

Training a neural network is typically viewed as a search for a single point in a high-dimensional parameter space. However, as models grow to include millions or billions of parameters, this "weight space" becomes increasingly difficult to navigate, communicate, and interpret. A growing body of research suggests that we should instead treat neural network weights as a structured data modality—much like images or text—that can be compressed into a low-dimensional latent manifold.

By learning these latent representations of weights, researchers are finding they can generate high-performing models without traditional training, quantify uncertainty more effectively, and compress models for edge devices.

The Weight Space Problem

The primary challenge in analyzing neural network parameters is their sheer dimensionality and the existence of "weight-space pathologies." Because of symmetries in neural architectures (like neuron permutations), many different sets of weights can represent the exact same mathematical function.

Melanie F. Pradier, Finale Doshi-Velez, and their colleagues addressed this in Latent Projection BNNs by introducing Proj-BNN. They argued that instead of performing inference in the massive, highly correlated weight space, we should learn a non-linear projection to a simpler latent space. This allows for more robust Bayesian inference, as the latent space bypasses the optimization hurdles of the raw parameter space.

From Structure to Function

A key insight across this field is that a "good" latent representation must capture more than just the numerical values of weights; it must capture the function the weights perform.

Standard autoencoders minimize the Euclidean distance between the original and reconstructed weights (structural loss). However, Ruikun Li and Huandong Wang in Predicting Dynamical Systems across Environments via Diffusive Model Weight Generation emphasize the use of functional loss. This ensures that the reconstructed weights produce the same output behavior as the original model. Similarly, Léo Meynent and Konstantin Schürholt demonstrate in Structure Is Not Enough that incorporating behavioral similarity into the compression process is essential for generating functional models from a latent distribution.

Generative Parameter Modeling

Once weights are mapped to a manageable latent manifold, we can use generative models to "hallucinate" new parameters. This shifts the paradigm from optimization (finding weights via SGD) to sampling (picking weights from a distribution).

  • Diffusion Models: Boya Zeng and Dongwen Tang introduced Neural Network Diffusion, which uses 1D convolutions to process flattened parameter vectors. By diffusing in the latent space, they can generate high-performing weights in seconds.
  • Task-Specific Generation: Bedionita Soro developed D2NWG, a framework that learns the distribution of weights from a "model zoo" to produce optimized parameters for new tasks, outperforming traditional transfer learning.
  • Policy Synthesis: In robotics, Yongyuan Liang and colleagues created Make-An-Agent, which uses a conditional diffusion model to generate policy weights directly from a single behavior demonstration.

Applications in Compression and Adaptation

The ability to represent a model as a small latent vector has profound implications for distributed systems and edge computing.

In Federated Learning, the communication of massive weight updates is a bottleneck. Minh-Duong Nguyen and others proposed HCFL, which treats weight updates as signals to be compressed by an under-complete autoencoder, drastically reducing bandwidth requirements.

For Large Language Models (LLMs), Yehui Tang and Kai Han introduced PocketLLM. By projecting weight subvectors into a discrete latent space and reconstructing them with lightweight decoders, they achieved 20× compression, enabling LLMs to run on resource-constrained devices.

The Path Forward: Meta-Representations

The ultimate goal of this research direction is the creation of "meta-representations"—abstractions that capture the underlying traits of a model family. Early foundations for this were laid by Stephen M. Omohundro in Family Discovery, and more recently by Theofanis Karaletsos, Peter Dayan, and Zoubin Ghahramani in Probabilistic Meta-Representations Of Neural Networks.

By embedding neurons or entire networks into a shared latent space, we move away from treating weights as independent variables. Instead, we view them as manifestations of a deeper, lower-dimensional functional manifold, allowing us to manipulate and generate intelligence with the same ease we currently manipulate images and text.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_latent_representations_of_neura_2443e4a6,
  title = {Learning Latent Representations of Neural Network Weights},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-latent-representations-of-neural-network-weights/rdr_2443e4a67a314e3e94e6e1df5e45be12},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_latent_representations_of_neura_2443e4a6_v1,
  title = {Learning Latent Representations of Neural Network Weights},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-latent-representations-of-neural-network-weights/rdr_2443e4a67a314e3e94e6e1df5e45be12/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}