Abstract
Recurrent neural networks (RNNs) are among the most promising artificial intelligence techniques, showing great potential for memory and linguistic understanding. Sophisticated RNNs like long short-term memory (LSTM) and gated recurrent units (GRUs) yield superior performance due to their core architectures. In this study, we developed a novel, compact architecture called YamRNN by stochastically generating and evaluating 30,000 RNN designs. YamRNN has significantly fewer parameters than LSTM and GRU (by a factor of two-thirds or better) and reduced training time by up to 80% on sequence classification tasks. This suggests significant room for developing more efficient architectures beyond human-designed standards.
Introduction
Recurrent Neural Networks (RNNs) are essential for processing sequential data, from speech recognition to financial forecasting. Currently, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs) dominate the field because their "gate" structures manage long-term dependencies effectively. However, these architectures are human-designed and complex. As computational efficiency becomes increasingly critical, the researchers investigated whether a more compact "core algorithm" exists within the vast topological space of possible RNN designs—one that might have been overlooked by traditional design approaches.
Methodology
The researchers employed a "bottom-up" stochastic search strategy to discover more efficient architectures:
- Stochastic Generation: 30,000 unique architectures were generated based on rules requiring a recurrent loop and at least two weight parameters.
- Compactness Constraints: To ensure efficiency, candidates were restricted to just two intermediate equations ($v_1$ and $v_2$), compared to the four used in LSTMs.
- Screening Process: Candidates underwent a two-stage screening. First, a 400-step memory task eliminated all but 579 candidates. Second, survivors were validated using Sequential MNIST.
- Final Selection: The top performer, YamRNN (Yamada modified RNN), was benchmarked against standard models (LSTM, GRU) and other compact variants.
Benchmarks included Sequential MNIST, S&P 500 stock movement prediction (100, 300, and 500-day sequences), IMDb sentiment analysis, and Reuters multi-class document classification.
Results and Discussion
Architectural Simplicity
YamRNN achieves its performance with a remarkably small footprint. It utilizes only three weight parameters and two bias parameters per unit.
This makes YamRNN roughly half the size of an LSTM and two-thirds the size of a GRU. The discovery of this unique topology suggests that human intuition may not be the optimal path for RNN design; exploring the topological space stochastically revealed a configuration that handles gradients more efficiently than traditional gates.
Learning Performance and Scalability
YamRNN demonstrated superior learning velocity, particularly as sequence lengths increased. In an 800-step memory benchmark, YamRNN converged approximately 150 times faster than LSTM and 9.9 times faster than GRU in terms of epochs.
On the S&P 500 dataset, YamRNN reduced computation time by 80% compared to industry standards when using 1024 units. Furthermore, while LSTM and GRU performance sometimes plateaued or deteriorated with increased unit counts, YamRNN’s performance improved consistently, reaching a peak accuracy of 0.970 on the SP500-100 task.
Conclusion
YamRNN represents a breakthrough in RNN efficiency, proving that ultra-compact models can outperform complex, human-designed architectures. While the search was limited to 30,000 candidates and two intermediate equations, the results indicate that further exploration of the RNN topological space using advanced search methods could yield even more powerful and efficient models. Future work will focus on the mathematical underpinnings of YamRNN's specific configuration to understand why its unique structure facilitates such rapid learning.
Summary updated