Abstract
This paper introduces EVOKE (EVOlution of systems with KErnel-based outputs), a hybrid machine learning architecture that integrates the temporal processing power of Recurrent Neural Networks (RNNs) with the robust classification capabilities of Support Vector Machines (SVMs). Traditional SVMs are static and struggle with long-term temporal dependencies, while gradient-based RNNs often fail due to the vanishing gradient problem. EVOKE addresses these limitations by using neuroevolution to optimize Long Short-Term Memory (LSTM) networks as temporal preprocessors, while employing SVMs to calculate optimal output mappings. The framework demonstrates exceptional generalization on context-sensitive grammar tasks and continuous time-series prediction.
Introduction
A fundamental challenge in sequence learning is managing long-term temporal dependencies. Support Vector Machines (SVMs), while powerful for static classification, typically rely on fixed-size input windows to process sequences, making them effectively blind to information outside that window. Conversely, Recurrent Neural Networks (RNNs) possess internal states that can theoretically store information indefinitely. However, training RNNs with standard gradient descent (e.g., Backpropagation Through Time) is notoriously difficult because error signals tend to vanish over long time lags. EVOKE was developed to combine the flexible, long-term memory of an RNN with the high-performance generalization of an SVM.
The EVOKE Architecture
The EVOKE framework, part of the broader Evolino (Evolution of Systems with Linear Outputs) family, consists of two cascaded modules designed to leverage the strengths of both evolutionary computation and kernel methods.
- The Preprocessor (RNN): An LSTM network serves as the temporal memory. Instead of using gradients, the weights are optimized using Enforced SubPopulations (ESP), an evolutionary algorithm. This approach avoids local minima and the vanishing gradient issues that plague gradient-based LSTMs.
- The Output Layer (SVM): The hidden states of the LSTM are treated as a high-dimensional feature space. These activations are fed into a standard SVM, which uses quadratic programming to find the optimal weights to map those states to the desired output.
The training loop involves evolving candidate LSTMs, collecting their internal activations on training sequences, and then training an SVM to map these activations to the target outputs. The fitness of an LSTM is determined by the SVM's performance on a validation set.
Experimental Evaluation
The authors evaluated EVOKE on two distinct types of problems:
- Context-Sensitive Grammar ($a^n b^n c^n$): A classic counting problem where the network must count 'a's and remember that count across long intervals to validate subsequent 'b's and 'c's.
- Multiple Superimposed Sine Waves: A continuous time-series task requiring the prediction of a signal composed of two non-harmonic sine waves ($sin(0.2x) + sin(0.311x)$). This is difficult because the signal never perfectly repeats.
Results and Discussion
EVOKE demonstrated superior generalization capabilities. On the $a^n b^n c^n$ task, after training on sequences with $n$ up to 10, the system generalized to $n=257$, significantly outperforming gradient-based LSTMs which only reached $n=29$.
For the sine wave task, EVOKE achieved a Mean Squared Error (MSE) of 0.021, outperforming Echo State Networks (ESNs). The system effectively learns an "adaptive kernel," where the evolved RNN projects sequences into a space that is linearly separable for the SVM.
Conclusion
EVOKE successfully combines the long-term memory of LSTMs with the generalization power of SVMs. By replacing gradient descent with neuroevolution, the framework bypasses common training pitfalls. While the SVM's kernel mapping can sometimes create a rugged fitness landscape, the overall approach provides a robust solution for complex sequence learning tasks. Future work includes co-evolving SVM hyperparameters and scaling the architecture for larger datasets.
Summary updated