Hierarchical Recurrent Neural Networks for Long-Term Dependencies
Lists
Only you see this
Abstract
We have already shown that extracting long-term dependencies from se(cid:173) quential data is difficult, both for determimstic dynamical systems such as recurrent networks, and probabilistic models such as hidden Markov models (HMMs) or input/output hidden Markov models (IOHMMs). In practice, to avoid this problem, researchers have used domain specific a-priori knowledge to give meaning to the hidden or state variables rep(cid:173) resenting past context. In this paper, we propose to use a more general type of a-priori knowledge, namely that the temporal dependencIes are structured hierarchically. This implies that long-term dependencies are represented by variables with a long time scale. This principle is applied to a recurrent network which includes delays and multiple time scales. Ex(cid:173) periments confirm the advantages of such structures. A similar approach is proposed for HMMs and IOHMMs.
BibTeX
@inproceedings{hihi1995hierarchical,
author = {Salah El Hihi and Yoshua Bengio},
title = {{Hierarchical Recurrent Neural Networks for Long-Term Dependencies}},
booktitle = {Advances in Neural Information Processing Systems},
year = {1995},
}| Learning long-term dependencies with gradient descent is difficult | IEEE Trans. Neural Networks · 1994 |
| Diffusion of Context and Credit Information in Markovian Models | arXiv · 1995 |
| Decoupling Hierarchical Recurrent Neural Networks With Locally Computable Losses | arXiv · 2019 |
| Character-Level Language Modeling with Hierarchical Recurrent Neural Networks | arXiv · 2016 |
| A Clockwork RNN | ICML · 2014 |
| Nested LSTMs | arXiv · 2018 |