Linear Least-Squares Based Methods for Neural Networks Learning

Oscar Fontenla-Romero, Deniz Erdogmus, José Carlos Príncipe, Amparo Alonso-Betanzos, Enrique F. CastilloICANN 2003 · DOI

TL;DR

This paper introduces two novel algorithms for supervised learning in feedforward neural networks: one for initialization and another for learning. Both methods leverage linear least squares to optimize subnetworks independently, effectively reducing the search space dimensionality and significantly decreasing the number of training epochs required to achieve good performance.

What Problem Does This Paper Address?

Training artificial neural networks, especially deep ones, can be notoriously slow and computationally expensive. Traditional gradient-based methods often struggle with convergence speed, requiring a large number of training epochs to find optimal weights. Furthermore, the initial choice of weights can heavily influence the training process and the final performance. This paper aims to accelerate this learning process by proposing methods that improve both the initial weight configuration and the subsequent learning updates, making neural network training more efficient.

Methods

The core innovation of this paper lies in reformulating parts of the neural network training problem as linear least-squares problems.

  • Linear Least-Squares Reformulation: The authors demonstrate that by using a specific cost function (related to the weighted Mean Squared Error) and applying a first-order Taylor series expansion, the minimization problem for a single layer of a neural network can be transformed into a linear system. This allows for the direct computation of optimal weights and biases for that layer, bypassing the iterative nature of gradient descent for that specific optimization step.
  • Initialization Algorithm: This algorithm leverages the linear least-squares reformulation to determine optimal initial weights and biases for each layer. It works by iteratively backpropagating a "desired signal" through the network, using the linear least-squares solution at each layer to set its initial parameters. This approach aims to provide a better starting point for the subsequent training process.
  • Learning Algorithm: This method proposes a hybrid approach. Initially, standard learning rules (like scaled conjugate gradient) are used. However, once the error reduction slows down, the algorithm switches to using linear least squares to optimize the last layer of the network, while the preceding layers continue to be updated with the standard method. This hybrid strategy aims to leverage the speed of linear least squares for the final layer's optimization.

Datasets

The paper uses a single dataset for evaluation:

  • K.U. Leuven prediction competition time series: This dataset is derived from a computer-simulated 5-scroll attractor, generated by a generalized Chua's circuit, and consists of 2000 samples. The desired signal was normalized to the interval [0.05, 0.95].

Key Insights and Ideas

  • Linearization for Efficiency: The most significant insight is that by carefully reformulating the cost function and utilizing approximations (first-order Taylor expansion), a non-linear optimization problem (neural network training) can be partially linearized. This allows for direct, non-iterative solutions for specific parts of the network.
  • Dimensionality Reduction: By optimizing subnetworks independently using linear least squares, the effective search space for the non-linear optimization is reduced. This is a key reason why fewer training epochs are needed.
  • Initialization Matters: The proposed initialization method, based on linear least squares, provides a better starting point than random initialization, leading to faster convergence and potentially better final solutions.
  • Hybrid Learning for Speed: Combining standard learning with linear least-squares optimization for the final layer offers a practical way to accelerate training, especially in the later stages when gradient descent might be slow.

Key Results

  • Initialization: The initialization method, when used with backpropagation, allowed training in 2000 epochs compared to 4000 epochs for random initialization, achieving similar or better final MSE.
  • Learning: The proposed hybrid learning method demonstrated faster convergence compared to the standard Scaled Conjugate Gradient (SCG) method on the K.U. Leuven time series data. Figure 4 visually shows the learning curves, where the proposed method reaches a lower error level in fewer epochs.

Limitations and Future Work

  • Approximation: The linearization relies on a first-order Taylor series expansion, which is an approximation. The paper acknowledges that this might not hold perfectly for all activation functions or network configurations.
  • Smoothness of Convergence: The initialization algorithm is noted as not having a "smooth convergence" and instead "jumps from one region to another of the weight space." This could potentially lead to instability in some scenarios.
  • Focus on Last Layer: The learning algorithm only optimizes the last layer using linear least squares. Future work could explore applying this technique to more layers or different parts of the network.
  • Generalizability: While the paper presents results on one dataset, further validation on a wider range of benchmarks and network architectures would be beneficial to fully assess the generalizability of the proposed methods.

Summary updated

BibTeX
@inproceedings{fontenlaromero2003linear,
  author = {Oscar Fontenla-Romero and Deniz Erdogmus and José Carlos Príncipe and Amparo Alonso-Betanzos and Enrique F. Castillo},
  title = {{Linear Least-Squares Based Methods for Neural Networks Learning}},
  booktitle = {ICANN},
  year = {2003},
  doi = {10.1007/3-540-44989-2\_11},
}
Figure