TL;DR
This paper proposes using the Extended Kalman Algorithm (EKA) to train multilayer perceptrons (MLPs), framing MLP training as a nonlinear dynamic system identification problem. EKA offers a significantly faster convergence than traditional back-propagation, especially for complex problems, by treating weight updates as a state estimation problem.
What Problem Does This Paper Address?
Training artificial neural networks, particularly multilayer perceptrons (MLPs), is a computationally intensive task. The dominant training algorithm, back-propagation, suffers from slow convergence, requiring thousands of iterations for complex problems like speech recognition. This slowness becomes a major bottleneck, hindering the practical application of MLPs to larger and more intricate tasks. The authors aim to find a more efficient training algorithm that can achieve faster convergence and handle more complex problems.
Methods
The core innovation of this paper is framing the training of MLPs as a nonlinear dynamic system identification problem. They leverage the Extended Kalman Algorithm (EKA), a well-established technique for estimating the state of nonlinear dynamic systems.
Here's how they adapt it:
- System State: The weights of the MLP are treated as the "state" of the dynamic system.
- Nonlinear Dynamic System: The MLP's forward pass, which maps inputs to outputs, is modeled as a nonlinear dynamic system.
- Observation: The desired output for a given input serves as the "observation" of the system.
- Kalman Filter Adaptation: The EKA recursively estimates the system's state (the weights) by incorporating new observations (training data). It linearizes the nonlinear system around the current state estimate to apply the Kalman filter equations.
This approach contrasts with back-propagation, which iteratively adjusts weights based on local gradient information. EKA, on the other hand, uses a global approach to estimate the optimal weights based on all available data up to that point.
Datasets
The paper does not use standard, named datasets. Instead, they create custom, artificially constructed examples for evaluation:
- Two-dimensional inputs: This allows for visualization of the decision regions learned by the networks.
- Two problem types:
- "Regions" problem: Involves mapping inputs to one of four distinct output classes, creating meshed and disconnected regions.
- "XOR" problem: A generalized 2-input XOR problem, where different quadrants form different regions.
- Training Set: For each example, a training set of 1000 random vectors was generated, uniformly filling the input region.
- Output Encoding: Outputs corresponding to a correct class were set to 0.9, and -0.9 otherwise.
Key Insights and Ideas
- MLP Training as System Identification: The paper's central insight is that training an MLP can be elegantly reframed as identifying the parameters (weights) of a nonlinear dynamic system. This perspective opens the door to applying powerful estimation techniques like the Kalman filter.
- EKA's Speed Advantage: The EKA consistently converges in significantly fewer iterations than back-propagation, especially for the tested problems. This is a crucial practical advantage for training complex networks.
- Global vs. Local Optimization: Back-propagation is a local optimization method, relying on gradient descent. EKA, by contrast, is a more global estimation technique, considering the entire history of observations to update its state estimate (weights). This might explain its robustness and faster convergence.
- Non-Local Nature of EKA: Unlike back-propagation's "delta rule," EKA's weight updates are propagated through the entire gradient matrix and covariance matrix, making it a non-local training algorithm.
Key Results
- Faster Convergence: For both the "Regions" and "XOR" problems, the EKA consistently converged in fewer iterations than back-propagation. For the "Regions" problem, the Kalman algorithm achieved a significantly lower RMS error in fewer sweeps.
- Improved Solutions: In the "Regions" problem, the Kalman algorithm obtained better solutions (as visualized in Figure 3) compared to back-propagation.
- Back-propagation Failure: For the "XOR" problem, back-propagation failed to converge to the desired solution in any of the tested configurations. The Kalman algorithm, while also struggling with very small networks (2 hidden nodes), was able to find solutions with more hidden nodes.
- Impact of Network Size: For the XOR problem, increasing the number of hidden nodes in the network led to smaller error values for both algorithms, but EKA still maintained a convergence advantage.
Limitations and Future Work
- Computational Complexity: The authors acknowledge that the Extended Kalman Algorithm is computationally complex. While it converges faster in terms of iterations, the computation per iteration might be higher than back-propagation.
- Non-Optimality for Nonlinear Systems: The paper notes that for nonlinear systems, the EKA filter is not strictly optimal, and the estimates can only be "loosely termed conditional means."
- Limited Examples: The evaluation was performed on artificially constructed, two-dimensional examples. The performance on larger, real-world datasets with higher dimensionality was not explored.
- Future Work: The paper implicitly suggests exploring the application of EKA to more complex, real-world problems and potentially investigating optimizations to mitigate its computational complexity.
Summary updated