Abstract
Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components - a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture.
Introduction
In the field of Natural Language Processing (NLP), LSTMs and GRUs are the standard workhorses for sequence modeling. To enhance model capacity, researchers typically increase hidden layer sizes or stack multiple layers hierarchically. However, stacking RNNs often incurs a "depth tax," where increasing layers makes feature propagation difficult and exacerbates vanishing or exploding gradient problems.
This paper introduces Recurrently Controlled Recurrent Networks (RCRN), a novel architecture designed to boost expressive power without the traditional headaches of deep stacking. RCRN suggests that horizontal cooperation between layers can be more effective than vertical hierarchy.
Methods: The Controller-Listener Framework
The core innovation of RCRN is the Controller-Listener architecture. Instead of stacking layers where the output of Layer 1 becomes the input of Layer 2, RCRN uses two components that work in tandem on the same "level":
- The Controller Cell: This consists of two separate RNNs. Their primary function is to compute the gating functions (specifically the forget and output gates) for the base model.
- The Listener Cell: This is the base RNN that processes the input sequence. Its internal state and final output are dynamically governed by the gates produced by the Controller.
By replacing standard linear gating functions with "expressive" gates learned by their own recurrent units, the model gains sophisticated control over how information is remembered or forgotten over time. To ensure practicality, the authors implemented CUDA-level optimizations, allowing the final recurrent operation to be parallelized across the dimension axis on the GPU.
Experimental Results
The authors conducted an extensive empirical study across 26 different datasets to demonstrate the general-purpose nature of RCRN, including sentiment analysis (SST, IMDb, Amazon), question classification (TREC), and entailment (SNLI).
Performance and Efficiency
The RCRN demonstrated superior performance across nearly every metric:
- Amazon Reviews: RCRN achieved the highest score on all 16 sub-datasets, consistently outperforming both standard BiLSTMs and 3-layer stacked versions.
- SST-5 & TREC: It reached 54.3% accuracy on SST-5 and a near state-of-the-art score of 96.2% on TREC.
- Computational Speed: Thanks to CUDA optimization, RCRN is significantly faster than a standard 3-layer BiLSTM. For sequence lengths of 256, the optimized RCRN was nearly 6.5x faster than a non-optimized 3-layer stack.
Conclusion
RCRN presents a compelling case for rethinking the design of "deep" recurrent models. By utilizing a controller-listener framework, the architecture avoids the bottleneck effects of hierarchical stacking while maintaining high parameter efficiency. Future work may involve integrating these recurrently controlled cells with modern attention mechanisms or testing their efficacy in generative tasks like Neural Machine Translation.
Summary updated