Recurrently Controlled Recurrent Networks

Yi Tay, Luu Anh Tuan, Siu Cheung Hui

Abstract

Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components - a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture.

Introduction

In the field of Natural Language Processing (NLP), LSTMs and GRUs are the standard workhorses for sequence modeling. To enhance model capacity, researchers typically increase hidden layer sizes or stack multiple layers hierarchically. However, stacking RNNs often incurs a "depth tax," where increasing layers makes feature propagation difficult and exacerbates vanishing or exploding gradient problems.

This paper introduces Recurrently Controlled Recurrent Networks (RCRN), a novel architecture designed to boost expressive power without the traditional headaches of deep stacking. RCRN suggests that horizontal cooperation between layers can be more effective than vertical hierarchy.

Methods: The Controller-Listener Framework

The core innovation of RCRN is the Controller-Listener architecture. Instead of stacking layers where the output of Layer 1 becomes the input of Layer 2, RCRN uses two components that work in tandem on the same "level":

  1. The Controller Cell: This consists of two separate RNNs. Their primary function is to compute the gating functions (specifically the forget and output gates) for the base model.
  2. The Listener Cell: This is the base RNN that processes the input sequence. Its internal state and final output are dynamically governed by the gates produced by the Controller.
The RCRN architecture diagram illustrating how the Controller Cell computes gates for the Listener Cell.

By replacing standard linear gating functions with "expressive" gates learned by their own recurrent units, the model gains sophisticated control over how information is remembered or forgotten over time. To ensure practicality, the authors implemented CUDA-level optimizations, allowing the final recurrent operation to be parallelized across the dimension axis on the GPU.

Experimental Results

The authors conducted an extensive empirical study across 26 different datasets to demonstrate the general-purpose nature of RCRN, including sentiment analysis (SST, IMDb, Amazon), question classification (TREC), and entailment (SNLI).

Performance and Efficiency

The RCRN demonstrated superior performance across nearly every metric:

  • Amazon Reviews: RCRN achieved the highest score on all 16 sub-datasets, consistently outperforming both standard BiLSTMs and 3-layer stacked versions.
  • SST-5 & TREC: It reached 54.3% accuracy on SST-5 and a near state-of-the-art score of 96.2% on TREC.
  • Computational Speed: Thanks to CUDA optimization, RCRN is significantly faster than a standard 3-layer BiLSTM. For sequence lengths of 256, the optimized RCRN was nearly 6.5x faster than a non-optimized 3-layer stack.
Experimental results across 16 Amazon review datasets, showing RCRN consistently outperforming BiLSTM and stacked BiLSTM baselines.

Conclusion

RCRN presents a compelling case for rethinking the design of "deep" recurrent models. By utilizing a controller-listener framework, the architecture avoids the bottleneck effects of hierarchical stacking while maintaining high parameter efficiency. Future work may involve integrating these recurrently controlled cells with modern attention mechanisms or testing their efficacy in generative tasks like Neural Machine Translation.

Summary updated

Abstract

Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components - a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture.
BibTeX
@inproceedings{tay2018recurrently,
  author = {Yi Tay and Luu Anh Tuan and Siu Cheung Hui},
  title = {{Recurrently Controlled Recurrent Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=rJZjP_bubS},
}
Figure