Graph Representations of Neural Network Parameters

32 concepts30 papers1 proposal

When we train, analyze, or manipulate deep neural networks, we traditionally treat their parameters as flat vectors or high-dimensional tensors. Standard optimization algorithms, hypernetworks, and meta-learning models process these weights as coordinate-wise arrays. However, this conventional view ignores a fundamental property of neural network architectures: their structural topology and inherent symmetries.

If you swap the positions of two hidden neurons in a fully connected layer and permute their corresponding incoming and outgoing weights, the mathematical function represented by the neural network remains completely unchanged. Yet, in a flat parameter vector, this permutation completely shuffles the values, forcing any downstream meta-model to learn $N!$ equivalent representations from scratch.

To overcome this limitation, a growing body of research models neural networks not as flat collections of numbers, but as graphs. By mapping neurons or filters to nodes and weights to edges, researchers can leverage network science and Graph Neural Networks (GNNs) to analyze, optimize, and generate neural network parameters in a way that naturally respects their underlying geometric and structural properties.


Mapping Neural Networks to Graphs

The core mechanism of this paradigm is the translation of a neural network's architecture into a graph representation. Depending on the specific architecture and the objective of the analysis, this mapping can take several forms.

Node-and-Edge Abstractions

In the most straightforward formulation, individual neurons serve as nodes, and the synaptic weights connecting them act as directed edges. For example, Lim et al. introduce Graph Metanetworks (GMNs), which represent neural networks as parameter-centric graphs where unique weights correspond to edges. This abstraction allows processing complexity to scale with the number of unique parameters rather than the total number of operations, making it highly efficient for architectures with extensive weight sharing. Similarly, Papa et al. apply graph-structured weight processing to Neural Field (NeF) representations, mapping weights and biases to edges and nodes so that message-passing architectures can learn directly from the internal connectivity of the neural field.

Standard View: Flat Parameter Vector
[ w1, w2, w3, w4, w5, w6 ]  --> Highly sensitive to neuron ordering

Graph View: Topological Network
   (Neuron Node A) --w1--> (Neuron Node C)
   (Neuron Node B) --w2--> (Neuron Node C)
  --> Permutation invariant; preserves structural context

Handling Complex Architectures

This graph-mapping strategy extends beyond simple multi-layer perceptrons:

  • Convolutional Neural Networks (CNNs): To avoid the computational explosion of mapping every single neuron in a spatial feature map, Vahedian et al. propose a "rolled" graph representation where entire convolutional filters serve as nodes, and edges are weighted by the norm of the kernel channels between them. Alternatively, Vartak and Hossain model CNN layers as bipartite graphs where nodes represent individual neurons and edges represent the connecting weights to isolate static parameter anomalies.
  • Kolmogorov-Arnold Networks (KANs): Because KANs place learnable functions on edges rather than fixed weights on nodes, Bar-Shalom et al. represent them as directed graphs where function coefficients are mapped directly to multi-dimensional edge features, preserving the functional relationships of the network.
  • Generative Hypernetworks: When generating parameters for a target model, Li et al. represent the target architecture as a directed multigraph, allowing hypernetworks to generate weights and biases as edges and nodes that strictly adhere to the target's topological layout.

Alternative Topological Views

The mapping can also be inverted or abstracted further depending on the analytical goal. Jiang et al. model neural networks as directed line graphs where the synaptic weights themselves are treated as nodes, allowing them to study how weights mutually influence one another during backpropagation. In theoretical physics frameworks, Dyer and Gur-Ari map the mathematical expectations of wide neural network outputs to graphical representations akin to Feynman diagrams, where vertices represent network components and edges represent weight covariance.


Why It Works: Symmetries and Inductive Biases

The primary reason graph representations outperform flat-vector representations when processing neural networks is that they bake the structural symmetries of the target network directly into the meta-model's architecture.

Permutation Invariance and Equivariance

When a meta-model processes a neural network graph using a Graph Neural Network (GNN), the GNN relies on message-passing operations. Because the aggregation of messages from neighboring nodes is permutation-invariant (e.g., using sum, mean, or max), the GNN will produce the exact same representation regardless of how the nodes are indexed or ordered. As Lim et al. demonstrate, respecting these internal neuron permutation symmetries allows graph metanetworks to generalize across diverse structures—such as Transformers, ResNets, and CNNs—where structure-agnostic baselines fail.

Scaling Symmetries

Neural networks exhibit symmetries beyond node permutations. For instance, in networks utilizing activation functions like ReLU, tanh, or sine, you can scale up the weights entering a neuron by a factor of $\alpha$ and scale down the outgoing weights by $1/\alpha$ without altering the network's overall output. Recognizing this, Kalogeropoulos et al. introduced Scale Equivariant Graph Metanetworks (ScaleGMNs). By designing message-passing operations that are equivariant to weight scaling, their framework significantly boosts performance when classifying, editing, or predicting the behavior of trained models.

Capturing Local Structural Context

Standard coordinate-wise optimizers treat every weight scalar as an independent entity. In contrast, representing parameters as edges in a graph allows a meta-model to capture local connectivity patterns and structural dependencies. As Knyazev et al. explain, this structural approach enables a meta-model to understand a parameter's functional role within the broader architecture, allowing it to predict parameter updates based on structural context rather than treating them as isolated scalars.


Applications of Neural Graph Representations

Transforming neural networks into graphs unlocks powerful tools from network science, geometric deep learning, and database querying, leading to breakthroughs across several domains.

Parameter Forecasting and Optimization

By treating a neural network as an evolving graph, meta-models can learn the general rules of weight evolution during training. Knyazev et al. developed a framework called NiNo (Neuron Interaction and Nowcasting) that uses GNNs to predict future parameter states based on their structural context, accelerating training by up to 50%. In scenarios involving continuous distribution shifts over time, Bai et al. model architectures as dynamic graphs within a recurrent Bayesian framework, updating parameters while ensuring that weight changes strictly respect the underlying connectivity of the network.

Analyzing Training Dynamics and Self-Organization

Graph representations allow researchers to peer inside the "black box" of learning by monitoring topological shifts:

  • Early Visualization: Early foundations by Wejchert and Tesauro mapped weight magnitudes and connectivity directly onto geometric primitives (bond diagrams) to inspect individual connection strengths in real time.
  • Phase Transitions: Shine et al. utilize asymmetric connectivity matrices to track how artificial networks reconfigure their internal structure across distinct topological phases during learning.
  • Performance Prediction: Vahedian et al. show that analyzing the structural evolution of a filter-level graph during the first few epochs of training can predict a model's final accuracy with up to 90% accuracy.
  • Geometric Embeddings: Moharil maps evolving weights as a sequence of graphs embedded within a hyperbolic space (a Poincaré ball), visualizing how deep models self-organize into hierarchical, functional modules.
  • Emergent Ability Metrics: Xiao models Large Language Models as weighted, directed graphs, applying multifractal analysis to neuron interactions to predict emergent capabilities (like reasoning) without running behavioral tests.
  • Hierarchy Uncovering: Vilalta et al. establish connections between feature nodes when weights exceed statistical thresholds, using community detection algorithms to uncover complex, instance-to-instance patterns in deep embeddings.

Security, Interpretability, and Querying

Representing weights as static or dynamic graphs makes structural anomalies and internal logic highly visible:

  • Backdoor Detection: Vartak and Hossain developed DeBUGCN, which treats a CNN's static weights as a bipartite graph and trains a Graph Convolutional Network to recognize the structural signatures of a trojan or backdoor attack without requiring any input data.
  • Policy Explainability: In reinforcement learning, Ao et al. represent an agent's evolving policy as a dynamic node-link graph, mapping physical connectivity to functional behavior to identify the emergence of internal logic.
  • Structural Databases: To formally interrogate neural network parameters, Grohe et al. introduced a framework for structural graph querying. By treating a neural network as a database of nodes and weighted edges, researchers can use declarative query languages to compute internal statistics and inspect structural properties that are completely inaccessible to purely functional, input-output evaluation methods.

Key Foundational Work

Neural Network Visualization (1989) — contributed 1 concept elements to this direction.
Building Graph Representations of Deep Vector Embeddings (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_graph_representation_of_neural_networks_d54820b3,
  title = {graph representation of neural networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/graph-representation-of-neural-networks/rdr_d54820b3dc5549b7896893460c5058d6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_graph_representation_of_neural_networks_d54820b3_v1,
  title = {graph representation of neural networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/graph-representation-of-neural-networks/rdr_d54820b3dc5549b7896893460c5058d6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake