Efficient Network Construction Through Structural Plasticity

Yu Cao, Xiaocong Du, Zheng Li, Yufei Ma, Yu Cao 0001IEEE J. Emerg. Sel. Topics Circuits Syst 2019 · DOI

TL;DR

This paper introduces Continuous Growth and Pruning (CGaP), a novel training method for deep neural networks that mimics the structural plasticity of the brain. By dynamically growing and pruning network components, CGaP achieves significantly more compact and efficient models with comparable or improved accuracy, outperforming traditional static network pruning methods.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) have achieved remarkable success across various domains, but their deployment on resource-constrained hardware (like embedded systems) is hindered by their massive computational requirements and large memory footprints. Traditional approaches to address this involve training an over-parameterized network and then pruning away redundant parts. However, this method is suboptimal as it introduces unnecessary complexity during training and results in fixed, potentially inefficient structures. Furthermore, these static structures struggle to adapt to dynamic tasks or lifelong learning scenarios. The paper aims to bridge this gap by proposing a more biologically inspired, dynamic approach to network construction that leads to inherently efficient and adaptable models.

Methods

The core of the proposed method is Continuous Growth and Pruning (CGaP), inspired by the structural plasticity observed in mammalian brains. Instead of starting with a large, fixed network, CGaP begins with a small "seed" network. The training process then involves two key phases:

  • Growth Phase: Periodically, CGaP identifies and adds "important" learning units (filters or neurons) based on a calculated saliency score. This score quantifies the impact of a unit on the loss function, effectively prioritizing units that contribute most to learning. This growth phase continues until a predefined "peak capacity" is reached.
  • Pruning Phase: Once the network reaches its peak capacity and has been trained, a pruning phase begins. This phase removes less important units, similar to traditional pruning, but it operates on a network that has already dynamically adapted its structure.

The saliency score is derived from the first-order Taylor expansion of the loss function with respect to the weights of a unit. This allows for a principled way to determine which units are most critical for performance. The growth and pruning are applied layer by layer, and the paper details specific mechanisms for mapping and initializing new units to ensure smooth integration.

Datasets

The paper evaluates CGaP on the following standard datasets:

  • MNIST
  • CIFAR-10
  • CIFAR-100
  • SVHN (Street View House Numbers)

Key Insights and Ideas

  • Biological Inspiration for Efficiency: Mimicking brain's structural plasticity (growth and pruning) offers a powerful paradigm for creating efficient DNNs, moving beyond static pruning.
  • Dynamic Structure is Key: A network that can dynamically grow and adapt its structure during training is more efficient and potentially more robust than one with a fixed, pre-defined architecture.
  • Saliency-Guided Growth: Using a saliency score to guide the addition of new network components is more effective than random growth, leading to lower loss and better learning.
  • Structured Sparsity for Hardware: CGaP naturally produces structured sparsity (entire filters/neurons are removed), which is crucial for achieving significant speedups and energy savings on hardware accelerators, unlike unstructured sparsity.
  • Trade-off Between Size and Accuracy: The method demonstrates that it's possible to achieve substantial model compression without sacrificing accuracy, and in some cases, even improving it.

Key Results

CGaP demonstrates significant improvements across various metrics and network architectures:

  • Model Compression: CGaP achieves substantial reductions in model size, for example, reducing parameters by 64.0% for ResNet-110 on CIFAR-10 and 90.4% for LeNet-5 on MNIST.
  • Computational Efficiency: The number of FLOPs is significantly reduced, with 32.5% reduction for ResNet-56 and 63.3% for ResNet-110 on CIFAR-10.
  • Accuracy: CGaP maintains or improves accuracy compared to baseline models and outperforms state-of-the-art pruning-only methods. For instance, on CIFAR-100, it achieved +0.37% accuracy improvement for VGG-19 with significant parameter reduction.
  • Hardware Efficiency (FPGA Simulation): CGaP shows substantial improvements in on-chip buffer access energy (e.g., 21.6% reduction for VGG-19 on CIFAR-100), DRAM access energy (e.g., 15% reduction for VGG-19 on CIFAR-100), and inference latency (e.g., 21.1% reduction for VGG-19 on CIFAR-100) compared to non-structured pruning methods.

Limitations and Future Work

  • Hyper-parameter Sensitivity: While the paper shows robustness to variations, CGaP still relies on several hyper-parameters (e.g., growth ratio, pruning rate, noise parameters) that require careful tuning. The paper provides empirical guidance for this tuning.
  • Dynamic Structure Visualization: The paper visualizes filter changes, but a more comprehensive understanding of how the dynamic structure evolves and its implications for learning could be further explored.
  • Lifelong Learning and Continual Learning: The authors suggest that the dynamic nature of CGaP makes it well-suited for lifelong and continual learning tasks, which would be a promising direction for future research.
  • Hardware Implementation: While FPGA simulations are presented, actual hardware implementation and validation on real-world devices would be a valuable next step.

Summary updated

BibTeX
@article{cao2019efficient,
  author = {Yu Cao and Xiaocong Du and Zheng Li and Yufei Ma and Yu Cao},
  title = {{Efficient Network Construction Through Structural Plasticity}},
  journal = {IEEE J. Emerg. Sel. Topics Circuits Syst},
  year = {2019},
  doi = {10.1109/jetcas.2019.2933233},
}
Figure