Abstract
Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap.
Introduction
Deep learning has traditionally relied on over-parameterized dense networks, leading to unsustainable environmental and financial costs. This is particularly problematic for edge devices with limited resources. While sparse neural networks offer a solution, they are typically created via pruning—training a massive dense model first, which is computationally wasteful.
Sparse training from scratch (sparse-to-sparse) aims to maintain sparsity throughout the lifecycle. However, existing methods face challenges: static structures often yield poor accuracy, methods like RigL require expensive dense gradient calculations, and random search methods like SET struggle in the vast search spaces of extreme sparsity.
The CTRE Approach
Inspired by synaptic plasticity and Hebbian theory—the idea that "neurons that fire together, wire together"—we propose Cosine similarity-based and Random Topology Exploration (CTRE). CTRE evolves the network's structure during training without ever needing dense gradients.
The algorithm follows a three-step cycle at each epoch:
- Standard Training: A forward and backward pass is performed on the current sparse structure.
- Pruning: A fraction ($\zeta$) of existing weights with the smallest magnitudes are removed.
- Growth: To replace pruned connections, CTRE calculates the cosine similarity between activation vectors of neurons in adjacent layers. High similarity indicates a potential functional relationship, and a new connection is "wired" between these neurons.
Two variants of the algorithm were developed:
- CTREseq: Employs similarity-based growth until accuracy plateaus, then switches to random exploration.
- CTREsim: Simultaneously combines similarity-based growth and random exploration in every epoch to avoid local minima.
Experimental Evaluation
CTRE was evaluated across eight diverse datasets, including tabular (Madelon, Isolet), image (MNIST, CIFAR-10/100), and text (PCMAC, BASEHOCK).
Key Findings
- Extreme Sparsity Performance: CTRE excels at densities below 0.5%. On the Madelon dataset at 0.3% density, CTRE achieved ~78.8% accuracy, significantly outperforming SET (61.9%).
- Efficiency and Convergence: The method achieves performance comparable to dense networks using 100x fewer connections and converges faster than competitors like RigL or SNIP.
- Activation-Based Growth: By using activations (cosine similarity) rather than gradients to evolve topology, CTRE avoids the memory overhead of dense-in-the-backward passes, making it ideal for low-resource hardware.
- Magnitude-Insensitive Metric: Cosine similarity acts as a robust proxy for potential importance, preventing the network from over-focusing on "loud" but uninformative neurons in noisy datasets.
Conclusion and Future Work
CTRE demonstrates that dynamic topology evolution based on biological principles can effectively train ultra-sparse networks. While currently focused on Multi-Layer Perceptrons (MLPs), future work will explore extending CTRE to Convolutional Neural Networks (CNNs) by adapting the similarity metric to spatial constraints. Additionally, realizing full hardware-level speedups will require the development of custom sparse kernels to move beyond the current use of masks in standard frameworks.
Summary updated