Sequential Adaptation of Radial Basis Function Neural Networks and its Application to Time-series Prediction

NeurIPS 1990

TL;DR

This paper introduces a novel sequential adaptation algorithm for Radial Basis Function (RBF) neural networks, called the "method of successive F-Projections." This algorithm efficiently learns from each data point, offering faster convergence and improved accuracy in time-series prediction compared to traditional stochastic approximation methods, particularly for chaotic systems.

What Problem Does This Paper Address?

Predicting time-series data, especially in nonstationary environments where the underlying generating model changes over time, is a significant challenge in signal processing and control systems. Traditional methods often require large amounts of data and can be slow to adapt. The paper addresses the need for efficient and rapid adaptation in neural networks for such tasks, aiming to improve prediction accuracy and convergence speed. Specifically, it focuses on applying these adaptive techniques to RBF networks, which are known for their ability to approximate complex functions.

Methods

The core of the paper is the application of the "method of successive F-Projections" to RBF neural networks. This method, previously developed for general function estimation, involves finding a posterior function estimate that is optimal in the least L2-norm sense while satisfying new constraints imposed by observations. For neural networks, this translates to an algorithm that:

  • Initializes network parameters: Either randomly or based on prior knowledge.
  • Updates parameters sequentially: For each new observation $(x_i, y_i)$, the algorithm finds the network parameters $\theta$ that minimize a cost function. This cost function includes two components:
  • Minimizing the difference between the network's prediction and the actual observation ($y_i$).
  • Minimizing the change in the network's parameters from the previous step, incorporating prior knowledge.
  • Uses an extended cost function: Unlike standard methods that only consider the current prediction error, the F-Projections method incorporates an integral term that reflects the prior parameter values. This term is crucial for leveraging existing knowledge and achieving faster convergence.
  • Handles noisy data: The method is adapted to accommodate noisy observations by relaxing the exact constraint to an inequality.

The paper contrasts this with a standard "stochastic approximation" (or LMS) method, which only uses the current prediction error in its cost function.

Datasets

The paper uses a deterministic chaotic series, specifically the logistic map, defined by the equation $z_n = 4z_{n-1}(1-z_{n-1})$. This series is generated and used for training and evaluating the RBF network's prediction capabilities. The authors do not introduce new benchmark datasets but rather use a well-known chaotic system to demonstrate the algorithm's effectiveness.

Key Insights and Ideas

  • Prior knowledge is powerful: The F-Projections method's inclusion of a term that considers prior parameter values significantly accelerates convergence and improves the accuracy of the learned mapping, even with limited training data.
  • Efficiency in adaptation: The algorithm learns from each observation efficiently, making it suitable for nonstationary environments where models change over time.
  • RBF networks are well-suited: The localized nature of RBF Gaussian nodes makes them stable under this adaptation scheme, as changes tend to be local, affecting only a few nodes.
  • Faster convergence for chaotic systems: The F-Projections algorithm demonstrates a much faster convergence rate in learning the underlying dynamics of a chaotic time-series compared to stochastic approximation.

Key Results

  • The RBF network trained with the F-Projections algorithm was able to map the underlying function of the logistic map chaotic series effectively.
  • Prediction error residuals showed a substantial reduction in the initial stages of training, with steady improvement as more samples were presented.
  • Figure 3 visually demonstrates that the F-Projections algorithm achieved a close-fit map of the underlying function after training on only 20 samples, while the stochastic approximation algorithm failed to construct a close-fit mapping even after 60 samples.
  • The F-Projections algorithm showed stability by maintaining its learned map up to 60 training samples.
  • The speed of convergence, measured by the number of training samples required, was significantly higher for F-Projections.

Limitations and Future Work

  • Deterministic noise-free conditions: The initial development of the F-Projection method was under deterministic noise-free conditions. While the paper discusses an adaptation for noisy data, the primary focus and evaluation are on a deterministic chaotic series.
  • Block estimation: The paper notes that the cost function developed for a recursive estimation algorithm can be applied to a block estimation method. This suggests potential for further exploration in how the algorithm can be integrated into different estimation frameworks.
  • Further exploration of a priori knowledge: The authors highlight that the cost term incorporating a priori knowledge can be used for training an already trained network upon receiving further information, indicating a direction for future research in continuous learning or model refinement.

Summary updated

BibTeX
@inproceedings{1990sequential,
  title = {{Sequential Adaptation of Radial Basis Function Neural Networks and its Application to Time-series Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1990},
}
Figure