Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks

Bagdat Myrzakhmetov, Rustem Takhanov, Zhenisbek AssylbekovCICLing 2019 · OpenReview

Abstract

In this paper we analyzed the dynamics of Recurrent Neural Network architectures. We explored the chaotic nature of state-of-the-art Recurrent Neural Networks: Vanilla Recurrent Network and Recurrent Highway Networks. Our experiments showed that they exhibit chaotic behavior in the absence of input data. We also proposed a way of removing chaos from Recurrent Neural Networks. Our findings show that initialization of the weight matrices during the training plays an important role, as initialization with the matrices whose norm is smaller than one will lead to the non-chaotic behavior of the Recurrent Neural Networks. The advantage of the non-chaotic cells is stable dynamics. At the end, we tested our chaos-free version of the Recurrent Highway Networks (RHN) in a real-world application. In the language modeling task, chaos-free versions of RHN perform on par with the original version.
BibTeX
@inproceedings{myrzakhmetov2019initial,
  author = {Bagdat Myrzakhmetov and Rustem Takhanov and Zhenisbek Assylbekov},
  title = {{Initial Explorations on Chaotic Behaviors of Recurrent Neural Networks}},
  booktitle = {CICLing},
  year = {2019},
  url = {https://openreview.net/forum?id=LtwgOUB9WH},
}

Loading related content...

Figure