Abstract
Recurrent Neural Networks (RNNs) have long been the dominant choice for sequence modeling but suffer from two major issues: difficulty in capturing very long-term dependencies and an inability to parallelize sequential computation. Consequently, non-recurrent models like the Transformer have gained popularity, using multi-head attention to capture global dependencies. However, these models often lack the necessary components to model local structures effectively and rely heavily on manually designed position embeddings. In this paper, we propose the R-Transformer, which combines the advantages of both RNNs and multi-head attention while avoiding their respective drawbacks. The R-Transformer effectively captures both local structures and global long-term dependencies without the use of position embeddings. Empirical results across a wide range of domains show that R-Transformer outperforms state-of-the-art methods by a large margin.
Introduction
For years, sequence modeling was dominated by RNNs such as LSTMs and GRUs. However, RNNs struggle with gradient vanishing or exploding over long distances and are slow to train due to their sequential nature. Transformers addressed these issues through multi-head attention, which allows for global dependency tracking and parallelization. Yet, Transformers treat every position in a sequence identically, ignoring natural local order (locality) unless position embeddings are added. These embeddings can be difficult to design and may not fully capture local nuances.
The R-Transformer seeks to bridge this gap by integrating a local recurrent mechanism into the Transformer architecture, enabling the model to capture local structures like an RNN while maintaining the global reach and parallelization of a Transformer.
Methods
The R-Transformer architecture consists of a stack of identical layers, each containing three hierarchical components:
- LocalRNN: Instead of processing the entire sequence with one RNN, a shared RNN is applied to small, overlapping local windows. This captures the sequential order and local structure of the data. Because these windows are processed independently, this step is fully parallelizable across the sequence.
- Multi-head Attention: This layer sits above the LocalRNN, allowing each position to attend to every other position in the sequence. This captures global, long-term dependencies without needing manual position embeddings.
- Position-wise Feedforward: A standard non-linear transformation layer used for final refinement.
These components are connected using residual connections and layer normalization, similar to the standard Transformer architecture, ensuring stable training and efficient gradient flow.
Experimental Evaluation
The authors evaluated the R-Transformer across several diverse domains to test its versatility and performance against established baselines like TCNs and vanilla Transformers.
Datasets
- Sequential MNIST: A long-range dependency task where 28x28 images are flattened into 784-step sequences. This tests the model's ability to maintain information over long distances.
- Nottingham Dataset: A collection of folk tunes used for polyphonic music modeling, evaluating the model's ability to handle complex temporal patterns.
- Penn Treebank (PTB): A classic benchmark for both character-level and word-level language modeling.
Key Results
The R-Transformer consistently outperformed standard Transformers, TCNs, and traditional RNNs:
- Music Modeling: On the Nottingham dataset, R-Transformer achieved a Negative Log-Likelihood (NLL) of 2.37, significantly outperforming the standard Transformer (3.34) and TCN (3.07).
- Language Modeling: In word-level language modeling on PTB, it achieved a perplexity of 84.38, compared to the Transformer's 122.37.
- Long-term Memory: On the sequential MNIST task, it reached 99.1% accuracy, proving its effectiveness in handling long-range dependencies better than a standard Transformer (98.2%).
Discussion
The empirical success of R-Transformer provides several insights into sequence modeling:
- Locality is Essential: In data like natural language or music, the immediate neighborhood of a token carries vital information that global attention mechanisms might otherwise dilute.
- RNNs are Effective at Small Scales: While RNNs fail on long sequences, they excel at modeling short-term sequential patterns. By limiting the RNN to a small window, the model bypasses the gradient vanishing problem.
- Natural Position Encoding: The sequential processing within the LocalRNN inherently encodes order, making the "hack" of adding position embeddings unnecessary.
- Efficiency through Independence: Because the LocalRNN operates on local windows independently, the model retains the high training speed and GPU-friendliness of the Transformer.
Conclusion
R-Transformer offers a powerful hybrid approach to sequence modeling, successfully combining the local structural awareness of RNNs with the global dependency tracking of Transformers. While future work is needed to explore its performance in sequence-to-sequence tasks like machine translation, its current results demonstrate a significant step forward in building more robust and efficient sequence models.
Summary updated