Learning continuous piecewise non-linear activation functions for deep neural networks

Xinchen Gao, Yawei Li, Wen Li, Lixin Duan, Luc Van Gool, Luca Benini, Michele Magno

Abstract

Activation functions provide the non-linearity to deep neural networks, which are crucial for the optimization and performance improvement. In this paper, we propose a learnable continuous piece-wise nonlinear activation function (or CPN in short), which improves the widely used ReLU from three directions, i.e., finer pieces, non-linear terms and learnable parameterization. CPN is a continuous activation function with multiple pieces and incorporates non-linear terms in every interval. We give a general formulation of CPN and provide different implementations according to three key factors: whether the activation space is divided uniformly or not, whether the non-linear terms exist or not, and whether the activation function is continuous or not. We demonstrate the effectiveness of our method on image classification and single image super-resolution tasks by simply changing the activation function. For example, CPN improves 4.78% / 4.52% top-1 accuracy over ReLU on MobileNetV2_0.25 / MobileNetV2_0.35 for ImageNet classification and achieves better PSNR on several benchmarks for super-resolution. Our implementation is available at https://github.com/xc-G/CPN.
BibTeX
@inproceedings{gao2023learning,
  author = {Xinchen Gao and Yawei Li and Wen Li and Lixin Duan and Luc Van Gool and Luca Benini and Michele Magno},
  title = {{Learning continuous piecewise non-linear activation functions for deep neural networks}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00315},
}

Loading related content...

Figure