Recurrently Controlled Recurrent Networks

Yi Tay, Anh Tuan Luu, Siu Cheung Hui

Introduction

Recurrent Neural Networks (RNNs), specifically LSTMs and GRUs, have long been the workhorses of sequence modeling. To make these models more powerful, researchers typically follow two paths: increasing the hidden dimension size or stacking multiple layers on top of each other.

However, both strategies have significant limitations. Increasing dimensions can lead to overfitting, while stacking layers introduces classic deep learning challenges like vanishing gradients and difficulty in propagating features through many vertical layers.

Is there a better way to utilize multiple RNN layers without simply stacking them? The Recurrently Controlled Recurrent Network (RCRN) proposes a novel architecture that uses one "controller" RNN to dynamically manage the gating functions of a "listener" RNN. By allowing multiple recurrent layers to operate jointly rather than hierarchically, RCRN provides a more expressive alternative to standard stacked LSTMs, consistently outperforming them across diverse NLP benchmarks.

The Controller-Listener Architecture

The core innovation of the RCRN is the shift from a vertical hierarchy to a horizontal "control" relationship where layers work together at the same level of the sequence.

The Recurrently Controlled Recurrent Network (RCRN) architecture, featuring a Controller Cell that dynamically manages the gating of a Listener Cell.
  • The Controller: This component consists of two separate RNN cells. Its primary job isn't to produce the final representation, but to learn the "logic" of the gates. It processes the input sequence and produces two hidden states ($h^1_t$ and $h^2_t$).
  • The Listener: This is a standard RNN that processes the input sequence. However, instead of using its own internal linear projections to decide what to forget or output, it "listens" to the controller.
  • Recurrent Gating: The controller’s hidden states are transformed into the forget gate and output gate for the listener. This means the gating mechanism itself is recurrent and has its own memory of the past, making the information flow much more dynamic than a standard LSTM.

To ensure this complex setup isn't prohibitively slow, the authors implemented CUDA-level optimizations. By using a custom kernel, they parallelized the gating operations across the dimension axis, making the RCRN roughly as fast as a standard 3-layer stacked BiLSTM.

Key Insights

  • Recurrence-over-Recurrence: The most innovative idea is making the gates themselves recurrent. In a standard LSTM, gates are "reactive" to the current input and the previous hidden state via a simple linear layer. In RCRN, the gates have their own deep temporal context provided by the controller.
  • Horizontal vs. Vertical Complexity: RCRN increases model capacity "horizontally." By having layers influence each other's internal compositionality at the same time step, the model avoids the signal attenuation often seen in deep "vertical" stacks.
  • Superiority over Stacking: An RCRN (which has roughly the same number of parameters as a 3-layer BiLSTM) consistently performs better than the 3-layer stack. This suggests that the way we combine recurrent layers matters as much as how many we have.

Experimental Results

The authors evaluated RCRN on a massive "potpourri" of 26 NLP datasets to prove its general-purpose utility, spanning Sentiment Analysis, Question Classification, Natural Language Inference, Answer Selection, and Reading Comprehension.

The RCRN demonstrated remarkable consistency, outperforming BiLSTMs and stacked BiLSTMs on all 26 datasets tested.

Performance comparison on Amazon Review datasets, demonstrating RCRN's consistent superiority over standard and stacked BiLSTMs.
  • Amazon Reviews: Across 16 domains, RCRN showed a macro average gain of +4% over standard BiLSTMs and +3.4% over 2-layer stacked BiLSTMs.
  • State-of-the-Art (SOTA): At the time of publication, RCRN achieved a new SOTA on the TREC question classification task with 96.2% accuracy.
  • Sentiment Analysis: On the SST-5 (Stanford Sentiment Treebank) dataset, RCRN reached 54.3%, outperforming complex models like the Bi-Attentive Classification Network (BCN).
  • Efficiency: Runtime analysis showed that while RCRN is more complex than a single-layer LSTM, its optimized version is nearly identical in speed to a 3-layer BiLSTM, proving it is a highly practical replacement for existing deep architectures.

Limitations and Future Work

While RCRN outperformed many attention-based models, the authors noted that for tasks like Natural Language Inference, adding cross-sentence attention still provides a boost. The RCRN is primarily an encoder improvement, not a strict replacement for attention mechanisms.

Additionally, the authors focused on encoding tasks and did not explore generative tasks like Neural Machine Translation (NMT) due to high GPU resource requirements. Future directions include exploring "Nested RCRNs" (controllers controlling other controllers) and testing the architecture in non-NLP domains like speech recognition or financial time-series, where complex temporal gating is crucial.

Summary updated

Abstract

Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components - a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture. Additionally, RCRN achieves state-of-the-art results on several well-established datasets.
BibTeX
@inproceedings{tay2018recurrently,
  author = {Yi Tay and Anh Tuan Luu and Siu Cheung Hui},
  title = {{Recurrently Controlled Recurrent Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJZjP_bubS},
}
Figure