Evolino for recurrent support vector machines

Jürgen Schmidhuber, Matteo Gagliolo, Daan Wierstra, Faustino J. Gomez

Abstract

This paper introduces Evoke (EVOlution of systems with KErnel-based outputs), a novel framework that integrates the sequence-processing capabilities of Recurrent Neural Networks (RNNs) with the robust classification precision of Support Vector Machines (SVMs). By employing an evolutionary algorithm to train a Long Short-Term Memory (LSTM) network as a dynamic preprocessor for an SVM, Evoke creates a "recurrent SVM." This architecture is capable of learning complex context-sensitive languages and non-linear time series that are beyond the reach of standard, static SVMs.

Introduction: The Challenge of Sequential Data

Support Vector Machines (SVMs) are renowned for their performance in classification and regression tasks. However, they are inherently static models, making them ill-suited for sequential data characterized by arbitrary or unknown time lags. Traditional approaches to adapt SVMs for time-series data—such as using fixed-size sliding windows or frequency-domain transformations—frequently fail when temporal dependencies exceed the chosen window size. Prior to this work, there was no established SVM-based mechanism capable of maintaining internal state representations to handle such arbitrary time lags.

Methods: The Evoke Architecture

Evoke addresses the limitations of static SVMs by cascading two distinct modules into a unified system:

  1. Recurrent Neural Network (RNN): A Long Short-Term Memory (LSTM) network serves as a dynamic feature extractor. It processes input sequences and maintains an internal state that captures temporal dependencies.
  2. Support Vector Machine (SVM): The SVM receives the internal activations ($\phi_n$) of the RNN as its input, producing the final prediction or classification.
Architecture of the Evoke system (a) and the internal LSTM cell structure (b).

Training via Neuroevolution

Because the SVM's input depends on the RNN's evolving internal states, standard gradient descent is computationally difficult. Instead, Evoke utilizes neuroevolution, specifically the Enforced SubPopulations (ESP) method, to evolve the LSTM weights. The training process follows a two-phase cycle:

  • Phase 1: Training sequences are passed through candidate RNNs. The resulting internal states are collected to train the SVM to map these states to target outputs.
  • Phase 2: The combined RNN+SVM system is evaluated on a validation set. The error serves as a "fitness" score for the evolutionary algorithm. High-performing RNNs are then mutated to generate the next population.

Experimental Evaluation

The authors evaluated Evoke on two distinct tasks designed to test discrete and continuous sequence processing:

  • Context-Sensitive Grammars ($a^n b^n c^n T$): A discrete task requiring the network to count symbols ('a', 'b', 'c') and remember these counts indefinitely to validate the string structure.
  • Multiple Superimposed Sine Waves: A continuous time-series prediction task involving frequencies that are not multiples of each other, creating complex, non-periodic waveforms.

Key Results and Generalization

Evoke successfully solved tasks that are theoretically impossible for standard SVMs. In the context-sensitive grammar task, Evoke demonstrated remarkable generalization capabilities. As shown in the results table, when trained on sequences with $n$ up to 10, Evoke generalized to $n=257$, significantly outperforming standard gradient-based LSTMs (G-LSTM).

Generalization performance on context-sensitive grammar tasks compared to G-LSTM and PI-Evolino.

In the continuous domain, Evoke predicted complex superimposed sine waves with an average summed squared error of 0.021. This is a significant result, as other state-of-the-art models like Echo State Networks (ESNs) often fail on this specific task.

Discussion and Limitations

The core innovation of Evoke is the conceptualization of the evolved LSTM as an adaptive kernel. It dynamically maps variable-length sequences into a high-dimensional space where the SVM can effectively define decision boundaries. By optimizing for validation performance through evolution, the system prioritizes generalization over simple training set memorization.

However, the study also identified limitations. Evoke was consistently outperformed by PI-Evolino, a simpler variant using a pseudoinverse linear layer instead of an SVM. For example, PI-Evolino generalized the grammar task to $n=374$. The authors suggest that the SVM's kernel mapping may create a "rugged" fitness landscape, making it more difficult for the evolutionary algorithm to find optimal weights compared to a linear output layer. Future research will explore co-evolving SVM kernel parameters alongside RNN weights to mitigate this issue.

Summary updated

BibTeX
@inproceedings{schmidhuber2006evolino,
  author = {Jürgen Schmidhuber and Matteo Gagliolo and Daan Wierstra and Faustino J. Gomez},
  title = {{Evolino for recurrent support vector machines}},
  booktitle = {ESANN},
  year = {2006},
}
Figure