SAS: Self-Augmentation Strategy for Language Model Pre-training

Yifei Xu, Jingqiao Zhang, Ru He, Liangzhu Ge, Chao Yang, Cheng Yang, Ying Nian Wu

Abstract

SAS (Self-Augmentation Strategy) simplifies language model pre-training by replacing the dual-network architecture of models like ELECTRA with a single, self-evolving network. By using its own predictions from previous epochs to generate contextualized training data, SAS achieves state-of-the-art performance on the GLUE benchmark while significantly reducing computational overhead and eliminating the complex balancing act required by generator-discriminator systems.

Introduction

In Natural Language Processing (NLP), pre-training is fundamental. Models like BERT revolutionized the field using Masked Language Modeling (MLM), while ELECTRA improved efficiency by introducing a separate "Generator" to replace words and a "Discriminator" to detect them (Replaced Token Detection, or RTD). However, ELECTRA requires training two separate networks and carefully balancing their capacities to prevent the generator from overpowering the discriminator.

SAS addresses these drawbacks by providing the benefits of contextualized data augmentation without the baggage of a two-network system. It consolidates the learning process into a single transformer encoder that serves as its own teacher.

Methods: The Self-Augmentation Strategy

The core innovation of SAS is Self-Augmentation. Instead of a separate generator, SAS uses a single transformer encoder equipped with two specialized heads: an MLM head and an RTD head. The training process follows a temporal loop where the model's state from a previous epoch informs the training data for the current one.

  • Epoch 1 (Cold Start): The model uses a simple frequency-based unigram distribution to replace tokens.
  • Epoch $t+1$: The model utilizes the MLM predictions it made in the previous epoch ($t$) to generate contextually plausible replacements for the current epoch.
  • Dual-Task Learning: In every forward pass, the model performs both MLM (predicting the original word) and RTD (detecting if a word was replaced).
The SAS architecture featuring a single Transformer encoder with dual MLM and RTD heads, utilizing model predictions from previous epochs for self-augmentation.

This creates a natural curriculum. Early in training, poor MLM predictions make the RTD task easy. As the model improves, its self-generated replacements become more contextually plausible, naturally increasing the difficulty of the RTD task and removing the need for manual capacity tuning.

Key Results

SAS was evaluated on the GLUE benchmark after pre-training on the English Wikipedia and BooksCorpus. It consistently outperformed predecessors across various model sizes while maintaining higher computational efficiency.

  • Small Models: SAS-Small achieved a GLUE score of 81.61, outperforming ELECTRA-Small (80.78) and BERT-Small (79.10).
  • Base Models: SAS-DA-Base (incorporating disentangled attention) achieved a GLUE score of 85.60, edging out ELECTRA-Base while utilizing fewer training FLOPs.
  • Efficiency: On a single V100 GPU, SAS-Small required approximately 24 hours for pre-training, a 31% reduction in wall-clock time compared to the 35 hours required for ELECTRA-Small.
Performance comparison of SAS-Base against other state-of-the-art models on the GLUE benchmark, showing superior results with lower training FLOPs.

Conclusion

SAS demonstrates that a single model can effectively serve as its own data augmentor. By sharing parameters between the MLM and RTD tasks and using a temporal feedback loop, SAS achieves superior performance with less computational effort than dual-network architectures. While current implementations rely on epoch-based dependencies, the strategy offers a promising path toward more efficient and self-scaling language model pre-training.

Summary updated

Abstract

The core of self-supervised learning for pre-training language models includes pre-training task design as well as appropriate data augmentation. Most data augmentations in language model pre-training are context-independent. A seminal contextualized augmentation was recently proposed in ELECTRA and achieved state-of-the-art performance by introducing an auxiliary generation network (generator) to produce contextualized data augmentation for the training of a main discrimination network (discriminator). This design, however, introduces extra computation cost of the generator and a need to adjust the relative capability between the generator and the discriminator. In this paper, we propose a self-augmentation strategy (SAS) where a single network is utilized for both regular pre-training and contextualized data augmentation for the training in later epochs. Essentially, this strategy eliminates a separate generator and uses the single network to jointly conduct two pre-training tasks with MLM (Masked Language Modeling) and RTD (Replaced Token Detection) heads. It avoids the challenge to search for an appropriate size of the generator, which is critical to the performance as evidenced in ELECTRA and its subsequent variant models. In addition, SAS is a general strategy that can be seamlessly combined with many new techniques emerging recently or in the future, such as the disentangled attention mechanism from DeBERTa. Our experiments show that SAS is able to outperform ELECTRA and other state-of-the-art models in the GLUE tasks with similar or less computation cost.
BibTeX
@inproceedings{xu2022sas,
  author = {Yifei Xu and Jingqiao Zhang and Ru He and Liangzhu Ge and Chao Yang and Cheng Yang and Ying Nian Wu},
  title = {{SAS: Self-Augmentation Strategy for Language Model Pre-training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21412},
}
Figure