Multiscale sequence modeling with a learned dictionary

Bart van Merriënboer, Amartya Sanyal, Hugo Larochelle, Yoshua BengioarXiv 2017 · arXiv

Introduction

Sequence modeling in Natural Language Processing (NLP) typically involves a trade-off between character-level and word-level representations. Character-level models offer maximum flexibility and avoid out-of-vocabulary (OOV) issues but struggle with long-range dependencies and vanishing gradients. Conversely, word-level models are computationally efficient and capture semantic compositionality but fail on rare words and morphologically rich languages. This paper introduces a hybrid multiscale sequence model that bridges this gap by learning a dictionary of multi-symbol tokens.

Methods

The proposed architecture utilizes a learned dictionary to optimize both flexibility and computational efficiency through two primary mechanisms.

Dictionary Learning

The model employs an adapted version of the Byte-Pair Encoding (BPE) algorithm. Unlike standard BPE, this approach includes a pruning step to remove tokens that become redundant or too rare as larger tokens are formed, maintaining a lean and effective vocabulary of sub-word units.

The Multiscale RNN

The core of the system is a Multiscale Recurrent Neural Network (RNN). Unlike standard models that process one character at a time, this model evaluates multiple possible tokens ending at any given position.

The multiscale RNN considers multiple token paths of varying lengths at each position, allowing for transitions over single characters or multi-character sub-word units.

To handle the variety of possible segmentations, the model marginalizes over all valid paths using a dynamic programming approach. Hidden states are averaged across incoming paths at each character position to maintain a consistent memory state regardless of the specific token processed.

Key Results

The multiscale model was evaluated on the Penn Treebank (PTB) and Text8 datasets, demonstrating superior performance and parameter efficiency.

On the Penn Treebank dataset, a multiscale model with a 2048-token dictionary achieved 1.42 bits per character (bpc), slightly better than a 1024-unit LSTM baseline (1.43 bpc). Notably, a smaller multiscale model with only 512 units achieved 1.41 bpc, outperforming the baseline while utilizing 51% fewer parameters.

Training curves on the Text8 dataset demonstrate that the Multiscale LSTM consistently achieves lower bits per character (BPC) than a regular LSTM throughout the training process.

On the Text8 dataset, the multiscale model reached 1.41 bpc, a significant improvement over the character-level baseline of 1.45 bpc. Qualitative analysis shows the model learns to treat common words and phrases as single units while retaining the ability to generate rare words character-by-character.

Conclusion

By integrating a learned dictionary into a multiscale RNN framework, this research demonstrates that it is possible to combine the robustness of character-level modeling with the efficiency of word-level approaches. The resulting models are more parameter-efficient and better at capturing long-term dependencies in text. Future work includes joint end-to-end learning of the dictionary and the language model.

Summary updated

Abstract

We propose a generalization of neural network sequence models. Instead of predicting one symbol at a time, our multi-scale model makes predictions over multiple, potentially overlapping multi-symbol tokens. A variation of the byte-pair encoding (BPE) compression algorithm is used to learn the dictionary of tokens that the model is trained with. When applied to language modelling, our model has the flexibility of character-level models while maintaining many of the performance benefits of word-level models. Our experiments show that this model performs better than a regular LSTM on language modeling tasks, especially for smaller models.
BibTeX
@misc{merrienboer2017multiscale,
  author = {Bart van Merriënboer and Amartya Sanyal and Hugo Larochelle and Yoshua Bengio},
  title = {{Multiscale sequence modeling with a learned dictionary}},
  year = {2017},
  eprint = {1707.00762},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.00762},
}
Figure