Abstract
Designing optimal Recurrent Neural Networks (RNNs) is a complex task due to the vast search space of nodes, memory cells, and connectivity structures. This study utilizes a neuro-evolutionary algorithm to systematically evaluate the performance of various RNN memory cells and "deep" recurrent connections (time skips) on real-world time series data. The findings suggest that simple neurons with deep recurrent connections often outperform complex, hand-crafted memory cells like LSTMs, and that the $\Delta$-RNN cell exhibits significant robustness compared to traditional gated cells.
Introduction
Modern RNNs typically rely on complex, hand-crafted memory cells such as LSTMs and GRUs to capture long-term dependencies. However, these cells' internal recurrent connections usually span only a single time step ($t-1$ to $t$). Historical literature suggests that "deep recurrent connections"—which skip multiple steps in time (e.g., $t-10$ to $t$)—may offer a more effective means of retaining long-term memory. This paper addresses the challenge of designing optimal RNNs by using neuro-evolution as an automated empirical analysis tool to compare the efficacy of different memory cells against deep recurrent connections.
Methodology
The researchers employed EXAMM (Evolutionary eXploration of Augmenting Memory Models), a neuro-evolutionary algorithm designed to grow RNN topologies and optimize their weights. The study involved training over 10.56 million RNNs across 5,280 experiments, totaling approximately 52,800 CPU hours.
The search space allowed the algorithm to select between simple neurons and various memory cells, including LSTM, GRU, MGU, UGRNN, and $\Delta$-RNN. Experiments were categorized into those allowing only standard 1-step recurrent connections and those permitting deep recurrent connections with time skips ranging from 1 to 10 steps. Training was conducted using Backpropagation Through Time (BPTT) and Stochastic Gradient Descent (SGD). To address vanishing gradients in unstructured evolved networks, a "gradient boosting" technique was introduced, which up-scales the gradient norm if it falls below a specific threshold, effectively acting as the inverse of standard gradient clipping (which was also employed for exploding gradients).
Datasets
The performance of the evolved models was evaluated on two complex, multivariate, non-seasonal real-world datasets:
- Aviation Flight Recorder Data (NGAFID): This dataset includes 10 flights of per-second data across 26 parameters, with the targets being aircraft Pitch and RPM.
- Coal-fired Power Plant Data: This dataset consists of 10 days of per-minute data from 12 burners across 12 parameters, targeting Main Flame Intensity and Supplementary Fuel Flow.
Results and Discussion
The empirical analysis yielded several key insights:
- Superiority of Deep Recurrence: The most significant improvements in generalization were attributed to deep recurrent connections rather than the complexity of memory cells. Time skips proved highly effective for sequence modeling.
- Performance of Simple Neurons: RNNs composed solely of simple neurons and deep recurrent connections (
simple+rec) performed remarkably well, challenging the assumption that complex gated cells are necessary for long-term temporal dependencies. - Robustness of $\Delta$-RNN: The $\Delta$-RNN cell consistently outperformed popular alternatives like LSTMs and GRUs, suggesting it as a reliable choice for time series forecasting.
- Inconsistency of GRUs: GRUs exhibited high variance in performance, frequently appearing in both the top and bottom tiers of results, indicating sensitivity to initialization or structural limitations.
Quantitatively, 11 of the top 12 performing RNNs utilized deep recurrent connections. The simple+rec configuration produced the best model for the Aviation Pitch dataset and the second-best for the Coal Flame dataset. While LSTMs performed adequately, they did not reach the top tier in the absolute best-case scenarios.
Conclusion
This study demonstrates that deep recurrent connections are a powerful tool for enhancing RNN performance, often surpassing the benefits of complex memory cells. The success of simple neurons with time skips and the robustness of the $\Delta$-RNN cell provide new directions for RNN architecture design. Future work will explore adaptive methods for selecting time delays and the development of new memory cells that intelligently gate information from multiple time delays.
Summary updated