Developing a Novel Recurrent Neural Network Architecture with Fewer Parameters and Good Learning Performance

Kazunori Yamada, Fangzhou Lin, Tsukasa Nakamura

Abstract

Recurrent neural networks (RNNs) are among the most promising artificial intelligence techniques, showing great potential for memory and linguistic understanding. Sophisticated RNNs like long short-term memory (LSTM) and gated recurrent units (GRUs) yield superior performance due to their core architectures. In this study, we developed a novel, compact architecture called YamRNN by stochastically generating and evaluating 30,000 RNN designs. YamRNN has significantly fewer parameters than LSTM and GRU (by a factor of two-thirds or better) and reduced training time by up to 80% on sequence classification tasks. This suggests significant room for developing more efficient architectures beyond human-designed standards.

Introduction

Recurrent Neural Networks (RNNs) are essential for processing sequential data, from speech recognition to financial forecasting. Currently, Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRUs) dominate the field because their "gate" structures manage long-term dependencies effectively. However, these architectures are human-designed and complex. As computational efficiency becomes increasingly critical, the researchers investigated whether a more compact "core algorithm" exists within the vast topological space of possible RNN designs—one that might have been overlooked by traditional design approaches.

Methodology

The researchers employed a "bottom-up" stochastic search strategy to discover more efficient architectures:

  1. Stochastic Generation: 30,000 unique architectures were generated based on rules requiring a recurrent loop and at least two weight parameters.
  2. Compactness Constraints: To ensure efficiency, candidates were restricted to just two intermediate equations ($v_1$ and $v_2$), compared to the four used in LSTMs.
  3. Screening Process: Candidates underwent a two-stage screening. First, a 400-step memory task eliminated all but 579 candidates. Second, survivors were validated using Sequential MNIST.
  4. Final Selection: The top performer, YamRNN (Yamada modified RNN), was benchmarked against standard models (LSTM, GRU) and other compact variants.

Benchmarks included Sequential MNIST, S&P 500 stock movement prediction (100, 300, and 500-day sequences), IMDb sentiment analysis, and Reuters multi-class document classification.

Results and Discussion

Architectural Simplicity

YamRNN achieves its performance with a remarkably small footprint. It utilizes only three weight parameters and two bias parameters per unit.

Parameter counts across various RNN architectures showing YamRNN's efficiency.

This makes YamRNN roughly half the size of an LSTM and two-thirds the size of a GRU. The discovery of this unique topology suggests that human intuition may not be the optimal path for RNN design; exploring the topological space stochastically revealed a configuration that handles gradients more efficiently than traditional gates.

Learning Performance and Scalability

YamRNN demonstrated superior learning velocity, particularly as sequence lengths increased. In an 800-step memory benchmark, YamRNN converged approximately 150 times faster than LSTM and 9.9 times faster than GRU in terms of epochs.

The number of epochs required for convergence as a function of input sequence length.

On the S&P 500 dataset, YamRNN reduced computation time by 80% compared to industry standards when using 1024 units. Furthermore, while LSTM and GRU performance sometimes plateaued or deteriorated with increased unit counts, YamRNN’s performance improved consistently, reaching a peak accuracy of 0.970 on the SP500-100 task.

Conclusion

YamRNN represents a breakthrough in RNN efficiency, proving that ultra-compact models can outperform complex, human-designed architectures. While the search was limited to 30,000 candidates and two intermediate equations, the results indicate that further exploration of the RNN topological space using advanced search methods could yield even more powerful and efficient models. Future work will focus on the mathematical underpinnings of YamRNN's specific configuration to understand why its unique structure facilitates such rapid learning.

Summary updated

Abstract

Recurrent neural networks (RNNs) are among the most promising of the many artificial intelligence techniques now under development, showing great potential for memory, interaction, and linguistic understanding. Among the more sophisticated RNNs are long short-term memory (LSTM) and gated recurrent units (GRUs), which emulate animal brain behavior; these methods yield superior memory and learning speed because of the excellent core structure of their architectures. In this study, we attempted to make further improvements in core structure and develop a novel, compact architecture with a high learning speed. We stochastically generated 30000 RNN architectures, evaluated their performance, and selected the one most capable of memorizing long contexts with relatively few parameters. This RNN, YamRNN, had fewer parameters than LSTM and GRU by a factor of two-thirds or better and reduced the time required to achieve the same learning performance on a sequence classification task as LSTM and GRU by 80% at maximum. This novel RNN architecture is expected to be useful for addressing problems such as predictions and analyses on contextual data and also suggests that there is room for the development of better architectures.
BibTeX
@inproceedings{yamada2020developing,
  author = {Kazunori Yamada and Fangzhou Lin and Tsukasa Nakamura},
  title = {{Developing a Novel Recurrent Neural Network Architecture with Fewer Parameters and Good Learning Performance}},
  booktitle = {Interdisciplinary},
  year = {2020},
  doi = {10.4036/iis.2020.r.01},
}
Figure