The Power of Approximating: a Comparison of Activation Functions

NeurIPS 1992

TL;DR

This paper investigates the approximation power of different activation functions in feedforward neural networks, focusing on how efficiently they can represent complex functions. The authors introduce a framework to compare activation functions based on their ability to approximate polynomials and binary thresholds, revealing that many seemingly different functions offer equivalent approximation capabilities under certain error bounds.

What Problem Does This Paper Address?

Neural networks are powerful function approximators, but their performance is heavily influenced by the choice of activation function. While it's known that networks with certain activation functions (like sigmoids) can approximate any continuous function, this paper delves deeper into the efficiency of these approximations. Specifically, it asks: how do different activation functions compare in terms of the size and depth of the neural network required to achieve a certain level of approximation accuracy? This is crucial for designing practical, resource-efficient neural network architectures. The authors aim to establish equivalences and hierarchies between various activation functions based on their approximation power, considering both analog and boolean function approximation.

Methods

The paper introduces a formal framework for comparing activation functions. They define "simulation" where one class of activation functions can mimic another with a bounded increase in network size and layers. Two key error models are considered: a "tight" error bound of $2^{-s}$ (where $s$ is network size) and a "relaxed" error bound of $s^{-d}$ (where $d$ is network depth).

A central concept is the "threshold-property," which characterizes activation functions capable of efficiently approximating binary thresholds. The authors also define "suitable" functions that can be represented by power series with specific coefficient properties, and "powerful" functions that combine suitability with the ability to approximate a fast-converging function. These definitions allow them to establish equivalences between classes of activation functions.

Datasets

Not discussed in the paper. The paper focuses on theoretical approximation capabilities rather than empirical evaluation on specific datasets.

Key Insights and Ideas

  • Equivalence Classes: Many seemingly different activation functions (e.g., sigmoid, rational functions, logarithms, exponentials, Gaussian, radial basis functions) are shown to be equivalent in their approximation power for the tight error bound $2^{-s}$. This suggests that the specific choice among these might be less critical than previously thought for certain approximation tasks.
  • The Importance of Threshold Approximation: For the relaxed error bound $s^{-d}$, the ability to efficiently approximate the binary threshold becomes the dominant factor in determining approximation power. This highlights the fundamental role of threshold-like behavior in computation.
  • Smoothness vs. Thresholding: While smooth functions like sigmoids are powerful for approximating analog functions, they also offer advantages in computing boolean functions due to their non-trivial smoothness, making them more efficient than simple threshold units in certain scenarios.
  • Splines as a Benchmark: Splines (with a single knot) and the standard sigmoid are shown to be equivalent for the tight error bound $2^{-s}$, even when splines are allowed to have degree $s$. This establishes splines as a strong benchmark for approximation power.
  • Generalization of Newman's Result: The paper generalizes Newman's approximation of the absolute value function $|x|$ by rational functions, showing that powerful functions can achieve this.

Key Results

  • Theorem 2.1: A large class of activation functions (including sigmoid, rational functions, logarithms, exponentials, Gaussian, and radial basis functions) are equivalent with respect to the error bound $e(s, d) = 2^{-s}$.
  • Theorem 2.2: Splines (of degree $s$ for nets of size $s$) are equivalent to the standard sigmoid with respect to $e(s, d) = 2^{-s}$.
  • Theorem 2.3: Linear splines and polynomial nets have a lower bound on their approximation error for specific functions (e.g., $x^2$, $|x|$), indicating that sigmoids are strictly more powerful for these cases.
  • Theorem 3.1: For the relaxed error bound $e(s, d) = s^{-d}$, activation functions with the "threshold-property" are equivalent to the standard sigmoid. This includes polynomial nets and nets of linear splines.
  • Theorem 4.1: Sigmoidal nets can compute certain boolean functions (like squaring) with a constant number of gates, whereas threshold nets require at least $\Theta(\log n)$ gates. This demonstrates sigmoids' superior efficiency for specific boolean tasks.

Limitations and Future Work

  • Lipschitz Bounds: The paper allows for exponentially large Lipschitz bounds in the nets during simulations. Future work could explore whether these bounds can be reduced.
  • Input Units: The paper notes that the equivalence of threshold-nets and sigmoidal nets for the relaxed error bound $s^{-d}$ holds if the number of input units is counted. The implications of disregarding input units are explored in the context of boolean functions.
  • Generalization of Smoothness: Theorem 4.1 can be generalized to separate threshold-nets from 3-times differentiable activation functions, but this smoothness requirement is more stringent than what might be necessary. Further investigation into minimal smoothness requirements for certain approximation powers could be fruitful.
  • Empirical Validation: While the paper is theoretical, future work could involve empirical studies to validate these theoretical equivalences and hierarchies on real-world tasks.

Summary updated

BibTeX
@inproceedings{1992power,
  title = {{The Power of Approximating: a Comparison of Activation Functions}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1992},
}
Figure