Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions

Jinhyeok Jang, Hyunjoong Cho, Jaehong Kim, Jaeyeon Lee, Seungjoon YangarXiv 2018 · arXiv

Introduction

In conventional deep learning, neurons within a single layer are typically treated as homogeneous units. They share the same activation function and compete equally to learn features from input data. While this approach is effective for general learning, it complicates model compression. When attempting to prune a network for deployment on resource-constrained edge devices, identifying redundant neurons often requires computationally expensive post-hoc analysis.

Researchers from ETRI and UNIST have proposed Deep Asymmetric Networks, a novel architecture that breaks this symmetry. By assigning unique sensitivities to individual neurons, the network is forced to prioritize and sort learned features by importance during training, creating an inherent roadmap for efficient pruning.

Methodology: Node-wise Variant Activation Functions

The fundamental innovation of this work is the Node-wise Variant Activation Function (NVAF). Unlike standard layers where every node $i$ uses the same function $f(u)$, each node in an asymmetric layer is assigned a scaled version:

$$f_i^l(u) = s_i^l f_0(u)$$

where $s_i^l$ is a sensitivity parameter for node $i$ in layer $l$.

Architecture of Deep Asymmetric Networks for (a) fully-connected and (b) convolutional layers.

Sensitivity Scaling and Gradient Dynamics

The sensitivity parameters are pre-defined and ordered such that $1 \ge s_1 \ge s_2 \ge \dots \ge s_n > 0$. This ordering introduces a "gradient hack": because weight updates in backpropagation are proportional to the slope of the activation function, neurons with higher sensitivity (smaller indices) learn faster and capture more significant features. To compensate for the slower learning of less sensitive neurons, the authors typically tripled the standard learning rate during training.

Different sensitivity scaling profiles (linear, exponential, and sigmoid-like) used to assign node-wise importance.

This mechanism mimics Principal Component Analysis (PCA) by incentivizing the network to "pack" critical information into lower-indexed neurons. Consequently, the network's knowledge is effectively sorted by index.

Pruning Strategy

Because features are pre-sorted, the pruning process becomes trivial. Instead of calculating complex importance metrics like $l_1$ or $l_2$ norms, the algorithm simply removes nodes starting from the highest index (the least sensitive). This "proactive" approach allows for aggressive pruning followed by a brief retraining phase to recover accuracy.

Experimental Results

The authors evaluated Deep Asymmetric Networks across several benchmarks, including MNIST, CIFAR-10, NTU RGB+D, and ImageNet.

  • CIFAR-10: An asymmetric CNN was pruned to just 18.13% of its original weights with no loss in accuracy (83.59% post-retraining). In contrast, removing 90% of nodes in a standard network before retraining caused accuracy to plummet to ~15-20%, while the asymmetric network maintained 53.25%.
  • Transfer Learning: The method is highly transferable. A pre-trained VGG-16 model on the CK+ dataset was fine-tuned with asymmetric activations and pruned to 24.48% of its size, actually improving accuracy from 96.30% to 97.51%.
  • Large-scale Models: A ResNet-50 model was successfully compressed to 16.82% of its original size while maintaining high performance. In some cases, entire convolutional blocks were pruned away, simplifying complex residual units into identity mappings.

Conclusion and Future Work

Deep Asymmetric Networks offer a powerful paradigm shift for model compression by making feature importance an intrinsic part of the training process. While the current implementation requires tuning the sensitivity decay profile and adjusting learning rates, it significantly simplifies the path to deploying high-performance DNNs on mobile and edge hardware. Future research may focus on hardware-aware sensitivity profiles to further optimize for specific architectural constraints.

Summary updated

Abstract

This work presents deep asymmetric networks with a set of node-wise variant activation functions. The nodes' sensitivities are affected by activation function selections such that the nodes with smaller indices become increasingly more sensitive. As a result, features learned by the nodes are sorted by the node indices in the order of their importance. Asymmetric networks not only learn input features but also the importance of those features. Nodes of lesser importance in asymmetric networks can be pruned to reduce the complexity of the networks, and the pruned networks can be retrained without incurring performance losses. We validate the feature-sorting property using both shallow and deep asymmetric networks as well as deep asymmetric networks transferred from famous networks.
BibTeX
@misc{jang2018deep,
  author = {Jinhyeok Jang and Hyunjoong Cho and Jaehong Kim and Jaeyeon Lee and Seungjoon Yang},
  title = {{Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions}},
  year = {2018},
  eprint = {1809.03721},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.03721},
}
Figure