Abstract
Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap.
Introduction
As deep learning models scale to billions of parameters, they become prohibitively expensive to train and deploy, especially on resource-constrained edge devices. Traditional "dense-to-sparse" methods (pruning) require training a full, massive model before cutting away the "fat," which defeats the purpose of saving resources during training.
Existing "sparse-to-sparse" methods, which maintain sparsity throughout the entire training process, have significant drawbacks. Some rely on random topology searches that struggle to find the right connections in vast search spaces, while others (like RigL) require calculating gradients for non-existent connections, which is computationally expensive. The challenge is: How can we efficiently "regrow" the most important connections in a sparse network without looking at the full gradient or relying on pure luck?
The CTRE Framework
The authors propose CTRE (Cosine similarity-based and Random Topology Exploration), which evolves the network topology during training. The process follows a cycle of feed-forward/back-propagation, pruning, and Hebbian-inspired regrowth.
Instead of using gradients for regrowth, the algorithm identifies new connections by calculating the cosine similarity between the activation vectors of neurons in adjacent layers. If two neurons show similar activation patterns across a batch of data, CTRE assumes they are related and "wires" them together, mimicking the Hebbian principle of "neurons that fire together, wire together."
Algorithmic Variants
Two variants of the algorithm are introduced to balance exploitation and exploration:
- CTREseq: Uses cosine similarity until validation accuracy plateaus, then switches to random search to escape local minima.
- CTREsim: Simultaneously uses both cosine similarity and random search in every epoch to balance exploitation of known patterns with exploration of new ones.
Experimental Evaluation
The algorithm was evaluated across eight diverse datasets, including tabular (Madelon, Isolet), image (MNIST, Fashion-MNIST, CIFAR-10/100), and text (PCMAC, BASEHOCK).
Key Insights
- The Power of Normalization: Cosine similarity (normalized dot product) is superior to simple dot products. It makes the algorithm "magnitude insensitive," preventing it from being distracted by "noisy" neurons with high activation values but low information content.
- Winning at Extreme Sparsity: CTRE shines most when the network is "extremely" sparse (e.g., less than 1% density). In these regimes, the similarity metric acts as a guided compass, finding informative paths much faster than random or gradient-based methods.
- Efficiency via Sampling: Using just a quarter of the training samples for similarity calculations yields nearly identical results, significantly reducing computational overhead.
Key Results
- Madelon Dataset: In a 3-layer MLP with only 0.3% density, CTRE achieved 78.8% accuracy, significantly outperforming the previous state-of-the-art (SET) at 61.9%.
- Convergence and Robustness: CTRE consistently found high-performing sub-networks faster than competitors like RigL and SNIP. Unlike gradient-based methods, which can fail when density drops below 0.3%, CTRE remains stable.
- Performance vs. Dense: It is possible to reach performance comparable to a dense network while using 100 times fewer connections, making it a prime candidate for "Green AI."
Conclusion
CTRE demonstrates that biologically inspired heuristics like Hebbian learning can effectively guide the evolution of sparse neural network topologies. While currently focused on Multi-Layer Perceptrons (MLPs), future work aims to adapt the similarity metric for Convolutional Neural Networks (CNNs) and further optimize the computational cost of similarity matrix calculations.
Summary updated