Traditional neural network pruning often treats weights as a collection of independent values. By ranking parameters by magnitude and snipping the smallest, we hope to retain the most "important" parts of the model. However, this local perspective ignores a fundamental truth: a neural network is a graph. If pruning accidentally severs all paths between two layers, the network suffers from "layer collapse," and the signal flow is permanently broken.
A growing body of research suggests that the success of a sparse model depends less on the specific weights retained and more on the preservation of the network’s underlying topological skeleton.
The Problem of Disconnected Paths
When we prune a network to extreme sparsity—say, removing 99% of its weights—the risk of creating "dead" neurons or entirely disconnected layers becomes a certainty for standard heuristics. Richard Schoonhoven and colleagues argue in LEAN that pruning must account for the interdependence between layers. Instead of looking at filters in isolation, they propose selecting complete multiplicative paths from input to output. This ensures that every preserved parameter is part of a functional, end-to-end circuit.
This path-centric view is echoed by Shreyas Malakarjun Patil and Constantine Dovrolis in their work on PHEW. They demonstrate that by using biased random walks to identify high-weight paths at initialization, one can construct sparse networks that are immediately trainable. By ensuring bidirectional connectivity—from input to output and vice versa—they prevent the "bottlenecks" that typically cause performance to collapse.
The Maximum Spanning Tree as a Functional Backbone
Why does simple magnitude pruning work at all? Recent topological analysis provides a surprising answer. Research into Iterative Magnitude Pruning (IMP) suggests that magnitude-based methods succeed because they implicitly preserve the Maximum Spanning Tree (MST) of the network's weight graph.
The MST represents the minimal set of strongest connections required to keep every neuron in a layer connected. A. Wang and Farzaneh Najafi leverage this insight to build biologically constrained RNNs. They show that by explicitly protecting this "topological skeleton," networks can maintain their information-processing capacity even at extreme sparsity. When the skeleton is preserved, the network retains a functional backbone that allows it to recover performance rapidly during fine-tuning.
Maintaining Information Flow
Beyond simple connectivity, the goal of preserving topology is to maintain the flow of information. Charles Westphal and colleagues propose Mutual Information Preserving Pruning, which ensures that the data's signal can be reconstructed even after the mask is applied. This provides a theoretical guarantee that the sparse network remains a functional approximator of the original dense model.
In the context of Large Language Models (LLMs), this structural integrity is even more critical. Yingtao Zhang and Lu Hou show in their Plug-and-Play method that the total removal of input or output channels is a primary driver of performance collapse. By using channel permutation to align significant connections, they preserve the underlying topology, allowing models like LLaMA2 to maintain reasoning capabilities at 50% sparsity without retraining.
Connectivity as a Regularizer
If connectivity is the goal, why not optimize for it directly? Christian Franssen and colleagues introduced CoNNect, a differentiable regularizer that treats the network as a graph and maximizes its connectivity during training. This approach prevents layer collapse by ensuring a functional signal path always exists, even when the pruning pressure is extreme.
Similarly, Hoang Pham and Shiwei Liu propose the Node-Path Balancing (NPB) principle in Towards Data-Agnostic Pruning At Initialization. They argue that the "best" sparse mask is one that balances the number of effective nodes with the number of complete paths. This global structural constraint outperforms methods that focus solely on local weight importance.
Conclusion
The shift from weight-wise pruning to topology-aware pruning marks a transition in how we understand neural network compression. By viewing the network as a graph and prioritizing the preservation of its topological skeleton—whether through path selection, MST protection, or connectivity regularization—we can move toward "sparsest models" that were previously thought to be unreachable. As Stephen Zhang and Vardan Papyan note, the most efficient models often elude us because we fail to validate path-connectivity, wasting parameter budgets on disconnected components. By focusing on the skeleton, we ensure that every remaining parameter actually contributes to the model's intelligence.