TL;DR
Researchers have developed a new "brain-inspired" algorithm called CTRE that trains highly sparse neural networks from scratch by evolving their topology based on neuron activation patterns. By using cosine similarity to identify important connections instead of expensive dense gradients, the method achieves state-of-the-art accuracy in extremely sparse regimes (less than 1% density) where traditional methods often fail.
What Problem Does This Paper Address?
Deep neural networks are notoriously resource-hungry. While "pruning" (removing unimportant weights) can create efficient sparse models, traditional methods usually require training a massive dense model first. This "dense-to-sparse" approach is a non-starter for low-power edge devices with limited memory.
Recent "sparse-to-sparse" methods try to train sparse networks from the get-go, but they face a dilemma:
- Random Regrowth: Adding connections randomly is computationally cheap but slow to converge.
- Gradient-based Regrowth: Using gradient information to find new connections is effective but requires calculating "dense" gradients for all possible connections, which defeats the purpose of sparse training.
Furthermore, existing methods often collapse when the network is pushed to extreme sparsity (e.g., 99.7% of connections removed), failing to find the critical information paths through the network.
Methods
The authors introduce Cosine similarity-based and Random Topology Exploration (CTRE). Inspired by Hebbian learning theory—the idea that "neurons that fire together, wire together"—CTRE evolves the network's structure during training.
The process follows a cycle:
- Standard Training: Perform a forward and backward pass on the current sparse weights.
- Magnitude Pruning: Remove a fraction ($\zeta$) of the smallest weights (a standard technique).
- Cosine Similarity Regrowth: To replace the removed weights, the algorithm calculates the cosine similarity between the activation vectors of neurons in adjacent layers. High similarity suggests a strong functional relationship.
- Random Exploration: The algorithm also incorporates random connection addition to prevent the topology from getting stuck in local minima.
The paper proposes two variants: CTREseq (switches from similarity to random search when accuracy plateaus) and CTREsim (dynamically balances both strategies in every epoch). Crucially, this method avoids calculating gradients for non-existing connections, making it much more efficient for sparse training.
Datasets
The authors evaluated CTRE across eight diverse benchmarks to prove its versatility:
- Tabular: Madelon (artificial/noisy), Isolet (speech).
- Image: MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
- Text: PCMAC and BASEHOCK (subsets of the 20 Newsgroups dataset).
Key Insights and Ideas
- Normalization is Key: While classic Hebbian learning uses the simple product of activations, CTRE uses cosine similarity. The authors found that normalization is vital, especially in noisy environments like the Madelon dataset. Without it, the algorithm gets "distracted" by noisy neurons that happen to have high activation magnitudes.
- The "Extreme Sparsity" Advantage: In very sparse regimes (densities < 0.5%), gradient-based methods like RigL often fail to find informative sub-networks. CTRE, by looking at the behavior of neurons (activations) rather than just the error signal (gradients), identifies important paths much more reliably.
- Fast Convergence: The authors introduced a "Training Delay" (TD) metric. They found that CTRE reaches high accuracy levels significantly faster than its competitors, which is critical for reducing energy consumption during training.
Key Results
- Superior Accuracy at High Sparsity: On the Madelon dataset with a 1000-neuron MLP at 0.3% density, CTRE reached 78.8% accuracy, while the next best method (SET) only managed 61.9%.
- Outperforming the Baseline: In many cases, CTRE-trained sparse networks outperformed their fully-connected (dense) counterparts, likely because the sparsity acted as a form of regularization against noise.
- Robustness: CTRE consistently ranked as the top or second-best performer across almost all tested datasets and sparsity levels, showing particular strength as the models grew larger and sparser.
Limitations and Future Work
- Computational Overhead: Calculating the similarity matrix for every layer adds an extra cost of $O(mN^l)$ per epoch (where $m$ is the number of samples). While the authors showed that using only a fraction of the samples (subsampling) can mitigate this without losing much accuracy, it remains an area for optimization.
- Extension to CNNs: The current work focuses on Multi-Layer Perceptrons (MLPs). Extending this to Convolutional Neural Networks (CNNs) is non-trivial because CNNs use weight sharing, a concept that doesn't have a direct biological equivalent in the Hebbian sense.
- Safety and Trade-offs: The authors note that while high sparsity saves energy, the potential drop in accuracy must be carefully weighed in safety-critical applications.
Overall, CTRE represents a significant step toward making deep learning more sustainable and accessible for resource-constrained hardware by mimicking the efficient, evolutionary wiring of the biological brain.
Summary updated