Abstract This paper reviews the basic scheme of the Pseudoinverse Learning (PIL) algorithm, a non-gradient descent and non-iterative learning method for multi-layer neural networks. First presented in 1995, PIL offers several advantages over traditional gradient descent algorithms, particularly in terms of computational speed and avoiding local minima. We present new viewpoints on the PIL algorithm and address common pitfalls in practical implementation. Furthermore, we demonstrate that the so-called Extreme Learning Machine (ELM) is a "Variant crEated by Simple name alTernation" (VEST) of the PIL algorithm for single hidden layer feedforward neural networks.
Introduction: Beyond Backpropagation
The dominant paradigm for training Multi-Layer Perceptrons (MLPs) has long been Backpropagation (BP). However, BP relies on gradient descent, which is notoriously slow, prone to getting stuck in local minima, and requires the tedious tuning of hyperparameters like learning rates. In the mid-1990s, the Pseudoinverse Learning (PIL) algorithm was proposed as a faster, analytical alternative. Instead of iteratively searching for weights, PIL calculates them in a single step using linear algebra, specifically the Moore-Penrose pseudoinverse.
The PIL Framework and Architecture
The core of the PIL algorithm is applied to a Single Hidden Layer Network (SHLN). The architecture consists of an input layer, a hidden layer, and an output layer, as illustrated below.
In this model, $V$ represents the weights connecting the input layer to the hidden layer, and $W$ represents the weights connecting the hidden layer to the output layer. The learning process for the output weights is defined by the equation:
$$W = Y^+ B$$
where $Y^+$ is the Moore-Penrose pseudoinverse of the hidden layer output matrix and $B$ is the target matrix. The paper discusses three primary methods for initializing the input weights ($V$):
- PIL: Setting $V$ as the pseudoinverse of the input data.
- PIL0: Randomly assigning $V$. This is the variant later rebranded as ELM.
- PIL1: Adding Gaussian noise to the weights to improve regularization.
The "VEST" Critique
A significant portion of this work is dedicated to clarifying the academic history of these algorithms. The author introduces the acronym VEST (Variant crEated by Simple name alTernation) to describe the practice of rebranding existing algorithms without proper attribution. By comparing the mathematical steps of the Extreme Learning Machine (ELM) with the PIL algorithm published nearly a decade earlier, the author demonstrates that ELM is essentially PIL0. This critique serves to restore recognition for PIL's priority in the field of fast-learning neural networks.
Theoretical Insights and Common Pitfalls
While PIL is efficient, the paper highlights several mathematical pitfalls often overlooked in the literature:
- The Invertibility Pitfall: Proponents of random-weight networks often claim that random weights always lead to an invertible hidden layer matrix. Using a synthetic 1D function as a counter-example, the author proves that without constraints, random weights can cause "activation function saturation." This leads to rank-deficient matrices that make the algorithm numerically unstable.
- The Hidden Node Dilemma: PIL can achieve zero training error by setting the number of hidden nodes ($l$) equal to the number of training samples ($N$). However, the author warns that this is a recipe for massive overfitting. Choosing the optimal $l$ remains a difficult model selection problem that neither PIL nor ELM automatically solves.
- Non-Differentiable Activation: A unique advantage of PIL is that it does not require activation functions to be differentiable. It only requires them to perform a nonlinear transformation that increases the rank of the weight matrix, providing more flexibility in network design compared to BP.
Conclusion
The Pseudoinverse Learning algorithm remains a powerful tool for rapid neural network training. While it is not a "silver bullet"—requiring careful attention to memory constraints and overfitting—it provides a robust alternative to gradient-based methods. This review underscores the importance of historical context in machine learning research and calls for the community to acknowledge the foundational role of PIL in the development of non-iterative learning schemes.
Summary updated