Tensor-based framework for training flexible neural networks

Yassine Zniyed, Konstantin Usevich, Sebastian Miron, David BriearXiv 2021 · arXiv

Introduction

Modern deep learning architectures typically rely on fixed activation functions (AFs), such as ReLU or Sigmoid, to introduce non-linearity. While effective, these rigid functions often necessitate deeper or wider networks to approximate complex mappings, leading to significant increases in parameter count and computational overhead. Zniyed et al. propose a novel tensor-based framework to learn Flexible Activation Functions (FAFs), allowing for the compression of complex subnetworks into single, highly expressive layers.

The Challenge of Rigid Activation Functions

Standard networks are constrained by the predefined shapes of their AFs. While "flexible" AFs—where the function's shape is learned during training—have been explored, they often present training difficulties. Previous methods for network simplification frequently relied solely on first-order derivatives (Jacobians), which often failed to capture the constant "bias" term of the function, leading to poor global approximations.

Proposed Framework: Flexible Activation Functions via Tensor Decomposition

The authors introduce a model where each neuron's activation is represented as a weighted sum of basis functions, such as polynomials or piecewise linear functions.

Architecture of a single layer with flexible nonlinearities $g_r$.

The core innovation lies in the use of Coupled Matrix-Tensor Factorization (CMTF). To address the bias problem, the framework fuses first-order information (a 3rd-order Jacobian tensor) with zeroth-order information (a matrix of actual output values). This dual-order approach allows for a more accurate estimation of the function's constant terms.

To solve the resulting optimization problem efficiently, the researchers employed a Constrained Alternating Least Squares (ALS) algorithm. This method updates variables iteratively while ensuring the learned functions remain within the specified basis constraints.

Experimental Evaluation

The framework was tested on a pretrained Convolutional Neural Network (CNN) for character classification using the ICDAR dataset. The researchers targeted the third layer of the CNN, which accounted for approximately 75% of the total parameters.

Examples of learned flexible activation functions using polynomial (bottom) and piecewise linear (top) bases.

Key results include:

  • Parameter Reduction: Replacing a heavy subnetwork with a single flexible layer reduced its parameters to 25% of the original size with an accuracy drop of less than 0.5%.
  • Depth Compression: Replacing two consecutive layers with one flexible layer resulted in only a 1.45% accuracy loss after fine-tuning, despite a massive reduction in network depth.
  • Methodological Superiority: The CMTF approach significantly outperformed tensor-only decomposition (CTD), validating the necessity of incorporating zeroth-order information for accurate function approximation.

Conclusion and Future Directions

This tensor-based framework demonstrates that flexible activation functions can serve as a powerful tool for neural network compression and simplification. By moving beyond fixed AFs and leveraging high-order tensor properties, the method achieves significant parameter efficiency. Future research will explore applying this technique directly to spatial convolution layers and investigating the theoretical identifiability of non-polynomial bases like piecewise linear functions.

Summary updated

Abstract

Activation functions (AFs) are an important part of the design of neural networks (NNs), and their choice plays a predominant role in the performance of a NN. In this work, we are particularly interested in the estimation of flexible activation functions using tensor-based solutions, where the AFs are expressed as a weighted sum of predefined basis functions. To do so, we propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. This technique fuses the first and zeroth order information of the NN, where the first-order information is contained in a Jacobian tensor, following a constrained canonical polyadic decomposition (CPD). The proposed algorithm can handle different decomposition bases. The goal of this method is to compress large pretrained NN models, by replacing subnetworks, {\em i.e.,} one or multiple layers of the original network, by a new flexible layer. The approach is applied to a pretrained convolutional neural network (CNN) used for character classification.
BibTeX
@misc{zniyed2021tensor,
  author = {Yassine Zniyed and Konstantin Usevich and Sebastian Miron and David Brie},
  title = {{Tensor-based framework for training flexible neural networks}},
  year = {2021},
  eprint = {2106.13542},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.13542},
}
Figure