Taking Notes on the Fly Helps Language Pre-Training

Qiyu Wu, Chen Xing, Yatao Li, Guolin Ke, Di He, Tie-Yan Liu

TL;DR

"Taking Notes on the Fly" (TNF) is a novel pre-training approach that accelerates language model training by up to 60% while simultaneously improving downstream performance. By maintaining a dynamic "note dictionary" of contextual information for rare words, TNF provides the model with enhanced semantic signals, effectively overcoming the training bottleneck caused by poorly optimized rare word embeddings.

The Challenge of Rare Words in Pre-Training

Unsupervised language pre-training (such as BERT or ELECTRA) has revolutionized Natural Language Processing, but it remains notoriously resource-intensive. Training a standard BERT-base model can take weeks on high-end hardware.

The authors identify a specific bottleneck hindering efficiency: the heavy-tail distribution of natural language. A substantial portion of words in any given corpus are "rare"—appearing only a handful of times. Because they are infrequent, their learned embeddings are often poorly optimized and noisy. In a masked language modeling task, if a rare word is crucial for predicting a missing token, the model struggles because the input signal is "blurry."

Conceptual illustration of how taking notes helps predict masked words.

This deficiency doesn't just impact the rare words themselves; it slows down the convergence of the entire Transformer architecture. Simply filtering out sentences containing rare words isn't viable, as they constitute a significant portion of the training data.

The "Taking Notes on the Fly" (TNF) Method

To address this, the authors propose Taking Notes on the Fly (TNF), a method inspired by how humans take notes when encountering new or difficult concepts to aid future recall.

Architecture of the TNF method, showing the interaction between the Transformer and the Note Dictionary.

The TNF framework operates through several key mechanisms:

  1. The Note Dictionary (NoteDict): The system maintains a dictionary where keys are rare words (defined empirically as words appearing 100–500 times in the corpus) and values are "notes"—historical contextual representations of those words gathered during training.
  2. Leveraging Notes: During the forward pass of pre-training, if the model encounters a rare word, it retrieves the corresponding note from the dictionary. The input to the Transformer then becomes a weighted average of the standard word embedding and this retrieved "note." This provides a much clearer semantic signal than relying solely on a noisy, under-trained embedding.
  3. Updating Notes: After the forward pass, the model calculates a new note for the word by mean-pooling the contextual representations of its neighbors in the current sentence. The dictionary is subsequently updated using an Exponential Moving Average (EMA), allowing the "note" to evolve smoothly as the model learns.
  4. Zero-Overhead Fine-tuning: Crucially, the note dictionary is discarded after pre-training. During fine-tuning on downstream tasks (like sentiment analysis or question answering), the model reverts to the standard Transformer architecture. This ensures TNF produces a better-trained model without introducing any memory or latency overhead during inference.

Experimental Setup

  • Pre-training Corpus: Models were trained on a combination of the English Wikipedia and BookCorpus (roughly 16GB of text), adhering to the standard BERT setup.
  • Evaluation Benchmark: The GLUE (General Language Understanding Evaluation) benchmark was used, encompassing eight diverse tasks such as CoLA (linguistic acceptability), RTE (entailment), and MRPC (paraphrase detection).

Key Insights

  • Efficiency via Better Data Utilization: TNF does not merely aim to improve rare word embeddings in isolation; it leverages them to make the training of the entire model more efficient. Clearer input signals facilitate faster gradient descent across all Transformer layers.
  • Cross-Sentence Signals: While standard Transformers process sequences independently, TNF effectively enables "cross-sentence" information transfer. Through the dictionary, the model "remembers" the contextual meaning of a rare word from a sentence processed thousands of steps prior.
  • The "Free Lunch" of Pre-training: Because the dictionary is utilized exclusively during the pre-training phase, TNF acts as a "plug-and-play" optimization that can be integrated into existing architectures like BERT and ELECTRA without inference penalties.

Results

  • 60% Speedup: TNF-equipped models achieve the same performance level as their backbone models (BERT or ELECTRA) in less than half the training time. For instance, BERT-TNF reached standard BERT's final performance in just 2 days compared to 5.7 days on identical hardware.
  • Superior Performance: When trained for an equivalent number of iterations (1 million steps), TNF consistently outperformed the base models across the GLUE benchmark.
  • Significant Gains on Small Data: The performance boost was particularly pronounced on tasks with limited training data, such as RTE (showing a 4.2 to 4.8 point improvement). This indicates that TNF provides a superior initialization point for subsequent fine-tuning.
  • Scalability: The methodology proved effective for both "Base" (110M parameters) and "Large" (335M parameters) model configurations.

Limitations and Future Work

  • Heuristic Thresholds: The criteria for defining "rare words" (100–500 occurrences) and the dictionary capacity (200k) rely on empirical heuristics specific to English. These parameters may require significant adjustment for other languages or specialized domains (e.g., medical or legal corpora).
  • Note Aggregation Strategies: Currently, notes are updated using simple mean pooling and EMA. Exploring more sophisticated methods for aggregating contextual information could yield further improvements.
  • Morphologically Rich Languages: The current study focuses on English. Adapting TNF for languages with complex morphology (like Turkish or Finnish) might necessitate alternative strategies for determining which linguistic "units" warrant notes.

Summary updated

Abstract

How to make unsupervised language pre-training more efficient and less resource-intensive is an important research direction in NLP. In this paper, we focus on improving the efficiency of language pre-training methods through providing better data utilization. It is well-known that in language data corpus, words follow a heavy-tail distribution. A large proportion of words appear only very few times and the embeddings of rare words are usually poorly optimized. We argue that such embeddings carry inadequate semantic signals, which could make the data utilization inefficient and slow down the pre-training of the entire model. To mitigate this problem, we propose Taking Notes on the Fly (TNF), which takes notes for rare words on the fly during pre-training to help the model understand them when they occur next time. Specifically, TNF maintains a note dictionary and saves a rare word's contextual information in it as notes when the rare word occurs in a sentence. When the same rare word occurs again during training, the note information saved beforehand can be employed to enhance the semantics of the current sentence. By doing so, TNF provides a better data utilization since cross-sentence information is employed to cover the inadequate semantics caused by rare words in the sentences. We implement TNF on both BERT and ELECTRA to check its efficiency and effectiveness. Experimental results show that TNF's training time is 60% less than its backbone pre-training models when reaching the same performance. When trained with same number of iterations, TNF outperforms its backbone methods on most of downstream tasks and the average GLUE score. Code is attached in the supplementary material.
BibTeX
@inproceedings{wu2021taking,
  author = {Qiyu Wu and Chen Xing and Yatao Li and Guolin Ke and Di He and Tie-Yan Liu},
  title = {{Taking Notes on the Fly Helps Language Pre-Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=cQcb2X4hQSE},
}
Figure