Learning continuous piecewise non-linear activation functions for deep neural networks
Lists
Only you see this
Abstract
Activation functions provide the non-linearity to deep neural networks, which are crucial for the optimization and performance improvement. In this paper, we propose a learnable continuous piece-wise nonlinear activation function (or CPN in short), which improves the widely used ReLU from three directions, i.e., finer pieces, non-linear terms and learnable parameterization. CPN is a continuous activation function with multiple pieces and incorporates non-linear terms in every interval. We give a general formulation of CPN and provide different implementations according to three key factors: whether the activation space is divided uniformly or not, whether the non-linear terms exist or not, and whether the activation function is continuous or not. We demonstrate the effectiveness of our method on image classification and single image super-resolution tasks by simply changing the activation function. For example, CPN improves 4.78% / 4.52% top-1 accuracy over ReLU on MobileNetV2_0.25 / MobileNetV2_0.35 for ImageNet classification and achieves better PSNR on several benchmarks for super-resolution. Our implementation is available at https://github.com/xc-G/CPN.
BibTeX
@inproceedings{gao2023learning,
author = {Xinchen Gao and Yawei Li and Wen Li and Lixin Duan and Luc Van Gool and Luca Benini and Michele Magno},
title = {{Learning continuous piecewise non-linear activation functions for deep neural networks}},
booktitle = {ICME},
year = {2023},
doi = {10.1109/icme55011.2023.00315},
}Loading related content...