Recurrent neural networks with trainable amplitude of activation functions

Su Lee Goh, Danilo P. MandicNeural Networks 2003 · DOI

TL;DR

This paper introduces a novel algorithm, AARTRL, that allows recurrent neural networks (RNNs) to adapt the amplitude of their activation functions in real-time. This makes RNNs more robust to signals with unknown and complex dynamics, leading to faster and more accurate predictions compared to standard training methods.

What Problem Does This Paper Address?

Recurrent Neural Networks (RNNs) are powerful tools for modeling sequential data, finding applications in areas like signal processing, prediction, and control. However, training RNNs, particularly for signals with rich and unpredictable dynamics, can be challenging. Traditional training algorithms like Real Time Recurrent Learning (RTRL) often require careful tuning of parameters and can struggle when the input signal's dynamic range is unknown. This paper addresses the limitation of fixed activation function amplitudes in RNNs, which can hinder their performance when dealing with signals whose characteristics are not well-defined beforehand.

Methods

The core contribution of this paper is the development of the Adaptive Amplitude Real Time Recurrent Learning (AARTRL) algorithm. This algorithm extends the standard RTRL by making the amplitude of the activation function trainable. Instead of a fixed amplitude, AARTRL allows this parameter to be adjusted during the learning process using a gradient descent approach. The authors explore three variations of AARTRL:

  • Common adaptive amplitude: A single amplitude parameter is shared across all neurons in the network.
  • Layer-wise adaptive amplitude: Different amplitude parameters are used for different layers of the network.
  • Neuron-wise adaptive amplitude: Each individual neuron has its own unique trainable amplitude.

The derivation involves modifying the RTRL update rules to incorporate the gradient of the error with respect to the activation function's amplitude.

Datasets

The paper evaluates the proposed AARTRL algorithm using synthetic datasets designed to mimic real-world signal characteristics:

  • Colored input: A synthetic signal generated using an AR(4) process, representing signals with correlated noise.
  • Nonlinear input: A synthetic signal generated using a specific nonlinear transformation of a previous signal, representing signals with complex, non-linear dynamics.
  • Speech signals: Used to demonstrate the adaptive amplitude's behavior on real-world, dynamic data.

Key Insights and Ideas

  • Adaptive amplitudes are crucial for unknown dynamics: The paper strongly suggests that allowing activation function amplitudes to adapt is essential for RNNs to effectively handle signals with unknown or changing dynamical characteristics.
  • Faster convergence: AARTRL consistently demonstrates faster error convergence compared to the standard RTRL algorithm, meaning the network learns more quickly.
  • Robustness to signal range: The ability to adapt amplitude means the RNN doesn't need prior knowledge of the input signal's dynamic range, making it more versatile.
  • Neuron-wise adaptation offers the best performance: While all AARTRL variants outperform RTRL, the case where each neuron has its own adaptive amplitude (Case 3) yields the best performance, albeit with increased complexity.

Key Results

The experimental results, visualized in Figures 2 and 3 and summarized in Table 1, highlight the effectiveness of AARTRL:

  • Improved Mean Squared Error (MSE): Across both nonlinear and colored input signals, all AARTRL variants achieved lower MSE than the standard RTRL.
  • Faster learning curves: Figures 2 and 3 show that the AARTRL algorithms (regardless of the specific variant) reach a lower error level and do so in fewer iterations than RTRL.
  • Table 1 Comparison:
    • RTRL with fixed amplitude: Average MSE of 0.0146 (nonlinear) and 0.0309 (colored).
  • RTRL with single trainable amplitude: Average MSE of 0.0118 (nonlinear) and 0.0120 (colored).
  • RTRL with layer-by-layer trainable amplitude: Average MSE of 0.0115 (nonlinear) and 0.0119 (colored).
  • RTRL with neuron-by-neuron trainable amplitude: Average MSE of 0.0110 (nonlinear) and 0.0116 (colored). This case shows the lowest MSE.

Limitations and Future Work

  • Increased complexity: The authors acknowledge that while neuron-wise adaptive amplitudes (Case 3) offer the best performance, they also introduce more parameters, leading to increased computational complexity and potential for overfitting.
  • Further investigation: The paper suggests that further research could explore more efficient ways to train individual amplitudes or investigate other adaptive mechanisms for RNNs. The authors also mention the potential for AARTRL in other applications beyond prediction.

Summary updated

BibTeX
@article{goh2003recurrent,
  author = {Su Lee Goh and Danilo P. Mandic},
  title = {{Recurrent neural networks with trainable amplitude of activation functions}},
  journal = {Neural Networks},
  year = {2003},
  doi = {10.1016/s0893-6080(03)00139-4},
}
Figure