Multiple-Weight Recurrent Neural Networks

Zhu Cao, Linlin Wang, Gerard de MeloIJCAI 2017 · DOI

TL;DR

This paper introduces "Multiple-Weight Recurrent Neural Networks" (MW-RNNs), a novel architecture that equips RNNs and LSTMs with multiple sets of weights. This allows the network to dynamically switch between different "modes" or "facets" of data, mimicking human cognitive flexibility and improving performance on complex, multi-faceted tasks.

What Problem Does This Paper Address?

Recurrent Neural Networks (RNNs), including popular variants like LSTMs and GRUs, have been incredibly successful in processing sequential data like text and speech. However, current architectures struggle with data that has a "multi-faceted" or "bimodal" distribution. Think about how humans can effortlessly switch between formal and informal language, or how a single piece of text might draw on different types of knowledge. Standard RNNs, with their single set of weights, tend to average out these distinct characteristics, hindering their ability to fully capture the nuances of such complex data. This paper aims to bridge this gap by enabling RNNs to adaptively handle these diverse data characteristics.

Methods

The core innovation is the introduction of Multiple-Weight Recurrent Neural Networks (MW-RNNs). Instead of a single weight matrix for operations like transforming inputs or updating states, MW-RNNs employ multiple weight matrices. During training, a gating mechanism learns to dynamically select and combine these matrices based on the current input and the network's internal state.

Specifically, the paper proposes replacing a standard operation like tanh(Wx + b) with a weighted sum of such operations using different weight matrices: ∑ pi * tanh(W_i*x + b_i). This mechanism is then integrated into existing RNN architectures like LSTMs and GRUs. For LSTMs, this results in a "Multi-Weight LSTM" where the cell state update can draw from multiple potential transformations. For GRUs, a similar adaptation is made. The weights pi for selecting these matrices are learned via a softmax function, conditioned on the input and state, allowing the network to adaptively choose the most appropriate "facet" for processing.

Datasets

The paper evaluates MW-RNNs on several tasks and datasets:

  • Language Modeling: Penn Treebank dataset.
  • Dialogue Systems: Restaurant Reservation Dialogs and Switchboard Dialog Act Corpus.
  • Cloze Tests: "Who-did-What" dataset.
  • Character-Level Word Embedding: Penn Treebank dataset.
  • Artificial Datasets: Created by combining English and Bio/News corpora, and English and French corpora, to specifically test the multi-faceted handling capabilities.

Key Insights and Ideas

  • Mimicking Human Adaptability: The core idea is to mimic the human ability to switch between different cognitive modes or contexts, which is crucial for handling complex, real-world data.
  • Dynamic Weight Selection: The learned gating mechanism allows the network to dynamically choose which set of weights to use, enabling it to adapt to different "facets" of the input data on a per-instance or even per-time-step basis.
  • Beyond Ensembles: Unlike ensemble methods that combine outputs of multiple independent models, MW-RNNs use a single network with a single internal state, but with multiple internal processing pathways.
  • Handling Bimodal Data: The architecture is particularly well-suited for data that exhibits bimodal or multimodal characteristics, where different subsets of the data might benefit from distinct processing strategies.

Key Results

The MW-RNN model consistently outperforms previous state-of-the-art models across various tasks:

  • Language Modeling: Achieved lower perplexity and higher ROUGE2 scores compared to GRUs and Multi-Function Recurrent Units (Mu-FuRUs). For example, it reduced perplexity by 21.3% over GRUs and improved ROUGE2 by 42%.
  • Dialogue Systems: Improved accuracy on the Restaurant Reservation Dialogs dataset from 48% to 51.3% and reduced perplexity on the Switchboard Dialog Act Corpus by 8.8%.
  • Cloze Tests: Achieved 5% higher accuracy than the best baseline on the "Who-did-What" dataset.
  • Character-Level Word Embedding: Outperformed a previous method by 0.8% absolute accuracy.
  • Analysis of Weight Selection: Experiments on artificial datasets showed that the model indeed learns to associate specific weight matrices with different types of input data (e.g., English vs. French sentences).

Limitations and Future Work

  • Error Analysis: The authors identified two main categories of failure in cloze tests: "Infrequent Mechanism" (errors due to unusual patterns not well-learned) and "Prior Knowledge" (errors requiring external world knowledge not present in the training data).
  • Future Directions: The paper suggests that overcoming "Infrequent Mechanism" errors might be addressed by more sophisticated inference mechanisms, while "Prior Knowledge" errors could potentially be mitigated by incorporating external dictionaries or encyclopedias. The authors also note that their framework is broadly applicable and can be extended to other RNN variants.

Summary updated

BibTeX
@inproceedings{cao2017multiple,
  author = {Zhu Cao and Linlin Wang and Gerard de Melo},
  title = {{Multiple-Weight Recurrent Neural Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/205},
}

Loading related content...

Figure