A Brain-inspired Algorithm for Training Highly Sparse Neural Networks

Zahra Atashgahi, Joost Pieterse, Shiwei Liu, Decebal Constantin Mocanu, Raymond Veldhuis, Mykola Pechenizkiy

TL;DR

Researchers have developed a new "brain-inspired" algorithm called CTRE that trains highly sparse neural networks from scratch by evolving their topology based on neuron activation patterns. By using cosine similarity to identify important connections instead of expensive dense gradients, the method achieves state-of-the-art accuracy in extremely sparse regimes (less than 1% density) where traditional methods often fail.

What Problem Does This Paper Address?

Deep neural networks are notoriously resource-hungry. While "pruning" (removing unimportant weights) can create efficient sparse models, traditional methods usually require training a massive dense model first. This "dense-to-sparse" approach is a non-starter for low-power edge devices with limited memory.

Recent "sparse-to-sparse" methods try to train sparse networks from the get-go, but they face a dilemma:

  1. Random Regrowth: Adding connections randomly is computationally cheap but slow to converge.
  2. Gradient-based Regrowth: Using gradient information to find new connections is effective but requires calculating "dense" gradients for all possible connections, which defeats the purpose of sparse training.

Furthermore, existing methods often collapse when the network is pushed to extreme sparsity (e.g., 99.7% of connections removed), failing to find the critical information paths through the network.

Methods

The authors introduce Cosine similarity-based and Random Topology Exploration (CTRE). Inspired by Hebbian learning theory—the idea that "neurons that fire together, wire together"—CTRE evolves the network's structure during training.

The process follows a cycle:

  1. Standard Training: Perform a forward and backward pass on the current sparse weights.
  2. Magnitude Pruning: Remove a fraction ($\zeta$) of the smallest weights (a standard technique).
  3. Cosine Similarity Regrowth: To replace the removed weights, the algorithm calculates the cosine similarity between the activation vectors of neurons in adjacent layers. High similarity suggests a strong functional relationship.
  4. Random Exploration: The algorithm also incorporates random connection addition to prevent the topology from getting stuck in local minima.

The paper proposes two variants: CTREseq (switches from similarity to random search when accuracy plateaus) and CTREsim (dynamically balances both strategies in every epoch). Crucially, this method avoids calculating gradients for non-existing connections, making it much more efficient for sparse training.

Datasets

The authors evaluated CTRE across eight diverse benchmarks to prove its versatility:

  • Tabular: Madelon (artificial/noisy), Isolet (speech).
  • Image: MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100.
  • Text: PCMAC and BASEHOCK (subsets of the 20 Newsgroups dataset).

Key Insights and Ideas

  • Normalization is Key: While classic Hebbian learning uses the simple product of activations, CTRE uses cosine similarity. The authors found that normalization is vital, especially in noisy environments like the Madelon dataset. Without it, the algorithm gets "distracted" by noisy neurons that happen to have high activation magnitudes.
  • The "Extreme Sparsity" Advantage: In very sparse regimes (densities < 0.5%), gradient-based methods like RigL often fail to find informative sub-networks. CTRE, by looking at the behavior of neurons (activations) rather than just the error signal (gradients), identifies important paths much more reliably.
  • Fast Convergence: The authors introduced a "Training Delay" (TD) metric. They found that CTRE reaches high accuracy levels significantly faster than its competitors, which is critical for reducing energy consumption during training.

Key Results

  • Superior Accuracy at High Sparsity: On the Madelon dataset with a 1000-neuron MLP at 0.3% density, CTRE reached 78.8% accuracy, while the next best method (SET) only managed 61.9%.
  • Outperforming the Baseline: In many cases, CTRE-trained sparse networks outperformed their fully-connected (dense) counterparts, likely because the sparsity acted as a form of regularization against noise.
  • Robustness: CTRE consistently ranked as the top or second-best performer across almost all tested datasets and sparsity levels, showing particular strength as the models grew larger and sparser.

Limitations and Future Work

  • Computational Overhead: Calculating the similarity matrix for every layer adds an extra cost of $O(mN^l)$ per epoch (where $m$ is the number of samples). While the authors showed that using only a fraction of the samples (subsampling) can mitigate this without losing much accuracy, it remains an area for optimization.
  • Extension to CNNs: The current work focuses on Multi-Layer Perceptrons (MLPs). Extending this to Convolutional Neural Networks (CNNs) is non-trivial because CNNs use weight sharing, a concept that doesn't have a direct biological equivalent in the Hebbian sense.
  • Safety and Trade-offs: The authors note that while high sparsity saves energy, the potential drop in accuracy must be carefully weighed in safety-critical applications.

Overall, CTRE represents a significant step toward making deep learning more sustainable and accessible for resource-constrained hardware by mimicking the efficient, evolutionary wiring of the biological brain.

Summary updated

Abstract

Sparse neural networks attract increasing interest as they exhibit comparable performance to their dense counterparts while being computationally efficient. Pruning the dense neural networks is among the most widely used methods to obtain a sparse neural network. Driven by the high training cost of such methods that can be unaffordable for a low-resource device, training sparse neural networks sparsely from scratch has recently gained attention. However, existing sparse training algorithms suffer from various issues, including poor performance in high sparsity scenarios, computing dense gradient information during training, or pure random topology search. In this paper, inspired by the evolution of the biological brain and the Hebbian learning theory, we present a new sparse training approach that evolves sparse neural networks according to the behavior of neurons in the network. Concretely, by exploiting the cosine similarity metric to measure the importance of the connections, our proposed method, Cosine similarity-based and Random Topology Exploration (CTRE), evolves the topology of sparse neural networks by adding the most important connections to the network without calculating dense gradient in the backward. We carried out different experiments on eight datasets, including tabular, image, and text datasets, and demonstrate that our proposed method outperforms several state-of-the-art sparse training algorithms in extremely sparse neural networks by a large gap. The implementation code is available on https://github.com/zahraatashgahi/CTRE
BibTeX
@article{atashgahi2022brain,
  author = {Zahra Atashgahi and Joost Pieterse and Shiwei Liu and Decebal Constantin Mocanu and Raymond Veldhuis and Mykola Pechenizkiy},
  title = {{A Brain-inspired Algorithm for Training Highly Sparse Neural Networks}},
  journal = {Machine Learning},
  year = {2022},
  doi = {10.1007/s10994-022-06266-w},
}
Figure