Simple Recurrence Improves Masked Language Models

Tao Lei, Ran Tian, Jasmijn Bastings, Ankur P. ParikharXiv 2022 · OpenReview

Abstract: In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into the Transformer. We compare our model to baselines following the training and evaluation recipe of BERT. Our results confirm that recurrence can indeed improve Transformer models by a consistent margin, without requiring low-level performance optimizations, and while keeping the number of parameters constant. For example, our base model achieves an absolute improvement of 2.1 points averaged across 10 tasks and also demonstrates increased stability in fine-tuning over a range of learning rates.

Introduction

The Transformer architecture has dominated Natural Language Processing (NLP) by relying almost entirely on attention mechanisms. While attention is powerful, it is inherently "memoryless," lacking the sequential inductive bias that characterized earlier Recurrent Neural Networks (RNNs). However, standard RNNs like LSTMs are difficult to scale on modern hardware because their sequential nature breaks the massive parallelism that makes Transformers efficient.

Researchers at Google have addressed this by introducing SwishRNN, a minimalist recurrent module designed to be both efficient and powerful. By replacing standard feed-forward layers with this lightweight recurrence, they achieved significant improvements in Masked Language Models (MLMs) like BERT, including a 2.1-point average improvement on downstream tasks and dramatically increased fine-tuning stability.

The SwishRNN Architecture

The core innovation is the replacement of the standard Feed-Forward Network (FFN) blocks in a Transformer layer with a SwishRNN block. Unlike a complex LSTM, SwishRNN uses only two matrix multiplications and a simple pooling-style recurrence. It utilizes the "Swish" activation function to gate information flow, deciding how much state to carry over from the previous hidden state versus the current input.

Comparison of the standard Transformer block (right) and the proposed SwishRNN block (left).

To overcome the sequential bottleneck of RNNs, the authors implemented the recurrence using the scan() function in TensorFlow and introduced a variable step size trick. This allows the model to process chunks of tokens at once in certain layers, reducing the number of sequential steps needed and keeping training speed within 20-30% of the standard Transformer.

Experimental Setup

The models were pre-trained on the standard Wikipedia + Book Corpus and evaluated on 10 tasks from the GLUE and SuperGLUE benchmarks. These tasks cover a wide range of linguistic challenges, including Natural Language Inference (MNLI, RTE), Question Answering (BoolQ, MultiRC), and Sentiment Analysis (SST2).

Key Results and Stability

The "Base" version of the SwishRNN model (12 layers) outperformed the standard BERT baseline by an average of 2.1 points across all tasks. While the gains were smaller on "Large" models (0.6 points), the improvement remained consistent.

One of the most significant findings was the model's robustness during fine-tuning. Standard BERT models are notoriously sensitive to learning rates; a sub-optimal choice often leads to failure in convergence. In contrast, SwishRNN models demonstrated remarkable stability.

Average performance on GLUE tasks across different learning rates, showing the superior stability of the SwishRNN model (Ours) compared to BERT baselines.

As shown in the stability plots, the SwishRNN model's performance remains high and flat even as the learning rate varies, whereas the baseline BERT performance drops off sharply. This suggests that the simple recurrence acts as a sophisticated dynamic pooling operator, allowing the model to aggregate information across sequences more effectively and reliably.

Conclusion and Future Work

SwishRNN demonstrates that adding a minimalist recurrent component to Transformers can yield substantial benefits in performance and training stability without requiring complex hardware optimizations. While there is a small "sequential tax" in training speed (20-30% overhead), the trade-off provides a more robust and capable model.

Future research may explore SwishRNN in encoder-decoder architectures like T5 or apply it to non-text domains, such as protein modeling, where capturing long-range sequential dependencies is critical.

Summary updated

Abstract

In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into the Transformer. We compare our model to baselines following the training and evaluation recipe of BERT. Our results confirm that recurrence can indeed improve Transformer models by a consistent margin, without requiring low-level performance optimizations, and while keeping the number of parameters constant. For example, our base model achieves an absolute improvement of 2.1 points averaged across 10 tasks and also demonstrates increased stability in fine-tuning over a range of learning rates.
BibTeX
@misc{lei2022simple,
  author = {Tao Lei and Ran Tian and Jasmijn Bastings and Ankur P. Parikh},
  title = {{Simple Recurrence Improves Masked Language Models}},
  year = {2022},
  eprint = {2205.11588},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.11588},
}
Figure