Introduction
Sequence modeling in Natural Language Processing (NLP) typically involves a trade-off between character-level and word-level representations. Character-level models offer maximum flexibility and avoid out-of-vocabulary (OOV) issues but struggle with long-range dependencies and vanishing gradients. Conversely, word-level models are computationally efficient and capture semantic compositionality but fail on rare words and morphologically rich languages. This paper introduces a hybrid multiscale sequence model that bridges this gap by learning a dictionary of multi-symbol tokens.
Methods
The proposed architecture utilizes a learned dictionary to optimize both flexibility and computational efficiency through two primary mechanisms.
Dictionary Learning
The model employs an adapted version of the Byte-Pair Encoding (BPE) algorithm. Unlike standard BPE, this approach includes a pruning step to remove tokens that become redundant or too rare as larger tokens are formed, maintaining a lean and effective vocabulary of sub-word units.
The Multiscale RNN
The core of the system is a Multiscale Recurrent Neural Network (RNN). Unlike standard models that process one character at a time, this model evaluates multiple possible tokens ending at any given position.
To handle the variety of possible segmentations, the model marginalizes over all valid paths using a dynamic programming approach. Hidden states are averaged across incoming paths at each character position to maintain a consistent memory state regardless of the specific token processed.
Key Results
The multiscale model was evaluated on the Penn Treebank (PTB) and Text8 datasets, demonstrating superior performance and parameter efficiency.
On the Penn Treebank dataset, a multiscale model with a 2048-token dictionary achieved 1.42 bits per character (bpc), slightly better than a 1024-unit LSTM baseline (1.43 bpc). Notably, a smaller multiscale model with only 512 units achieved 1.41 bpc, outperforming the baseline while utilizing 51% fewer parameters.
On the Text8 dataset, the multiscale model reached 1.41 bpc, a significant improvement over the character-level baseline of 1.45 bpc. Qualitative analysis shows the model learns to treat common words and phrases as single units while retaining the ability to generate rare words character-by-character.
Conclusion
By integrating a learned dictionary into a multiscale RNN framework, this research demonstrates that it is possible to combine the robustness of character-level modeling with the efficiency of word-level approaches. The resulting models are more parameter-efficient and better at capturing long-term dependencies in text. Future work includes joint end-to-end learning of the dictionary and the language model.
Summary updated