A VEST of the Pseudoinverse Learning Algorithm

Ping Guo

Abstract This paper reviews the basic scheme of the Pseudoinverse Learning (PIL) algorithm, a non-gradient descent and non-iterative learning method for multi-layer neural networks. First presented in 1995, PIL offers several advantages over traditional gradient descent algorithms, particularly in terms of computational speed and avoiding local minima. We present new viewpoints on the PIL algorithm and address common pitfalls in practical implementation. Furthermore, we demonstrate that the so-called Extreme Learning Machine (ELM) is a "Variant crEated by Simple name alTernation" (VEST) of the PIL algorithm for single hidden layer feedforward neural networks.

Introduction: Beyond Backpropagation

The dominant paradigm for training Multi-Layer Perceptrons (MLPs) has long been Backpropagation (BP). However, BP relies on gradient descent, which is notoriously slow, prone to getting stuck in local minima, and requires the tedious tuning of hyperparameters like learning rates. In the mid-1990s, the Pseudoinverse Learning (PIL) algorithm was proposed as a faster, analytical alternative. Instead of iteratively searching for weights, PIL calculates them in a single step using linear algebra, specifically the Moore-Penrose pseudoinverse.

The PIL Framework and Architecture

The core of the PIL algorithm is applied to a Single Hidden Layer Network (SHLN). The architecture consists of an input layer, a hidden layer, and an output layer, as illustrated below.

Architecture of a Single Hidden Layer Network (SHLN) used in the PIL algorithm.

In this model, $V$ represents the weights connecting the input layer to the hidden layer, and $W$ represents the weights connecting the hidden layer to the output layer. The learning process for the output weights is defined by the equation:

$$W = Y^+ B$$

where $Y^+$ is the Moore-Penrose pseudoinverse of the hidden layer output matrix and $B$ is the target matrix. The paper discusses three primary methods for initializing the input weights ($V$):

  1. PIL: Setting $V$ as the pseudoinverse of the input data.
  2. PIL0: Randomly assigning $V$. This is the variant later rebranded as ELM.
  3. PIL1: Adding Gaussian noise to the weights to improve regularization.

The "VEST" Critique

A significant portion of this work is dedicated to clarifying the academic history of these algorithms. The author introduces the acronym VEST (Variant crEated by Simple name alTernation) to describe the practice of rebranding existing algorithms without proper attribution. By comparing the mathematical steps of the Extreme Learning Machine (ELM) with the PIL algorithm published nearly a decade earlier, the author demonstrates that ELM is essentially PIL0. This critique serves to restore recognition for PIL's priority in the field of fast-learning neural networks.

Theoretical Insights and Common Pitfalls

While PIL is efficient, the paper highlights several mathematical pitfalls often overlooked in the literature:

  • The Invertibility Pitfall: Proponents of random-weight networks often claim that random weights always lead to an invertible hidden layer matrix. Using a synthetic 1D function as a counter-example, the author proves that without constraints, random weights can cause "activation function saturation." This leads to rank-deficient matrices that make the algorithm numerically unstable.
  • The Hidden Node Dilemma: PIL can achieve zero training error by setting the number of hidden nodes ($l$) equal to the number of training samples ($N$). However, the author warns that this is a recipe for massive overfitting. Choosing the optimal $l$ remains a difficult model selection problem that neither PIL nor ELM automatically solves.
  • Non-Differentiable Activation: A unique advantage of PIL is that it does not require activation functions to be differentiable. It only requires them to perform a nonlinear transformation that increases the rank of the weight matrix, providing more flexibility in network design compared to BP.

Conclusion

The Pseudoinverse Learning algorithm remains a powerful tool for rapid neural network training. While it is not a "silver bullet"—requiring careful attention to memory constraints and overfitting—it provides a robust alternative to gradient-based methods. This review underscores the importance of historical context in machine learning research and calls for the community to acknowledge the foundational role of PIL in the development of non-iterative learning schemes.

Summary updated

Abstract

In this paper, we briefly review the basic scheme of the pseudoinverse learning (PIL) algorithm and present some discussions on the PIL, as well as its variants. The PIL algorithm, first presented in 1995, is a non-gradient descent and non-iterative learning algorithm for multi-layer neural networks and has several advantages compared with gradient descent based algorithms. Some new viewpoints to PIL algorithm are presented, and several common pitfalls in practical implementation of the neural network learning task are also addressed. In addition, we show that so called extreme learning machine is a Variant crEated by Simple name alTernation (VEST) of the PIL algorithm for single hidden layer feedforward neural networks.
BibTeX
@misc{guo2018vest,
  author = {Ping Guo},
  title = {{A VEST of the Pseudoinverse Learning Algorithm}},
  year = {2018},
  eprint = {1805.07828},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1805.07828},
}
Figure