Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks

Chang Xu, Weiran Huang, Hongwei Wang, Gang Wang, Tie-Yan Liu

Introduction

Traditional Recurrent Neural Networks (RNNs) process text as a uniform sequence of tokens, treating every word connection with equal weight. However, natural language is hierarchical and structured: words like "at," "all," and "costs" form a single semantic unit (a phrase), whereas "document" and "at" might be neighbors but share little semantic correlation. Capturing these local dependencies is challenging due to the diversity of phrase patterns and their discrete, symbolic nature, which makes them difficult to learn via standard backpropagation without external tools.

The Multi-Channel Recurrent Neural Network (MC-RNN) addresses these issues by using a staggered, multi-block design to explicitly capture local semantic structures. By employing an attention mechanism to dynamically select the best connection pattern at each time step, the model significantly improves performance in machine translation, summarization, and language modeling—especially for long-range dependencies.

Architecture and Methodology

The core innovation of the MC-RNN is its multi-channel, multi-block architecture designed to discover local structures automatically.

The MC-RNN architecture featuring staggered blocks across multiple channels and a multi-channel attention module.

Multi-Channel Multi-Block Design

The model maintains several parallel channels, each divided into "blocks" of a fixed size ($n$). Within a block, every node is fully connected to its predecessors, allowing a hidden state to directly "see" multiple previous words. Crucially, the blocks in different channels are staggered (offset by one step). This ensures that any potential phrase of length $\le n$ is contained within a single block in at least one of the channels.

Multi-Channel Attention Module

Since different channels represent different local dependency patterns, the model requires a mechanism to choose the optimal one. At each time step, an attention module calculates a weight for each channel based on the current input and hidden states. The final output is a weighted average of all channels, allowing the model to dynamically focus on the channel that best captures the current local context.

Experimental Results

The authors evaluated MC-RNN across three major NLP benchmarks: IWSLT 2014 German-to-English translation, Gigaword abstractive summarization, and Penn Treebank language modeling.

Performance comparison between Baseline-RNN and MC-RNN across different sentence lengths, showing significant gains for long sentences.

Machine Translation and Long-Range Dependencies

In the IWSLT 2014 task, MC-RNN achieved a BLEU score of 32.23, outperforming the Baseline-RNN (31.03). As shown in the figure above, the most striking improvement occurred on sentences longer than 61 tokens, where MC-RNN outperformed the baseline by over 4.0 BLEU points. This suggests that the block structure effectively mitigates the vanishing gradient problem by providing "shortcuts" between distant nodes.

Summarization and Language Modeling

On the Gigaword summarization task, the model achieved a gain of +1.90 ROUGE-1 points over the baseline. In language modeling on the PTB corpus, MC-RNN reached a test perplexity of 56.9, setting a new state-of-the-art for models without post-processing enhancements like continuous cache pointers.

Discussion and Insights

The MC-RNN transforms the discrete problem of phrase discovery into a continuous optimization task via attention weights. This structural parameterization allows the model to learn grammar-like structures through standard training. Furthermore, the flexible topology adapts to specific sentence contexts, unlike previous higher-order RNNs with fixed connections.

While the model introduces some computational overhead due to its multi-channel nature, the channels can be computed in parallel, making it practical for real-world applications. Future work may involve integrating these local dependency mechanisms into self-attention frameworks like Transformers, which have largely superseded recurrent architectures in recent years.

Summary updated

Abstract

Recurrent Neural Networks (RNNs) have been widely used in processing natural language tasks and achieve huge success. Traditional RNNs usually treat each token in a sentence uniformly and equally. However, this may miss the rich semantic structure information of a sentence, which is useful for understanding natural languages. Since semantic structures such as word dependence patterns are not parameterized, it is a challenge to capture and leverage structure information. In this paper, we propose an improved variant of RNN, Multi-Channel RNN (MC-RNN), to dynamically capture and leverage local semantic structure information. Concretely, MC-RNN contains multiple channels, each of which represents a local dependence pattern at a time. An attention mechanism is introduced to combine these patterns at each step, according to the semantic information. Then we parameterize structure information by adaptively selecting the most appropriate connection structures among channels. In this way, diverse local structures and dependence patterns in sentences can be well captured by MC-RNN. To verify the effectiveness of MC-RNN, we conduct extensive experiments on typical natural language processing tasks, including neural machine translation, abstractive summarization, and language modeling. Experimental results on these tasks all show significant improvements of MC-RNN over current top systems.
BibTeX
@inproceedings{xu2019modeling,
  author = {Chang Xu and Weiran Huang and Hongwei Wang and Gang Wang and Tie-Yan Liu},
  title = {{Modeling Local Dependence in Natural Language with Multi-Channel Recurrent Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33015525},
}
Figure