SAS: Self-Augmentation Strategy for Language Model Pre-training

Yifei Xu, Jingqiao Zhang, He Ru, Liangzhu Ge, Chao Yang, Cheng Yang, Ying Wu

Abstract

The core of self-supervised learning for pre-training language models includes pre-training task design as well as appropriate data augmentation. Most data augmentations in language model pre-training are context-independent. A seminal contextualized augmentation was recently proposed in ELECTRA and achieved state-of-the-art performance by introducing an auxiliary generation network (generator) to produce contextualized data augmentation for the training of a main discrimination network (discriminator). This design, however, introduces extra computation cost of the generator and a need to adjust the relative capability between the generator and the discriminator. In this paper, we propose a self-augmentation strategy (SAS) where a single network is utilized for both regular pre-training and contextualized data augmentation for the training in later epochs. Essentially, this strategy eliminates a separate generator and uses the single network to jointly conduct two pre-training tasks with MLM (Masked Language Modeling) and RTD (Replaced Token Detection) heads. It avoids the challenge to search for an appropriate size of the generator, which is critical to the performance as evidenced in ELECTRA and its subsequent variant models. In addition, SAS is a general strategy that can be seamlessly combined with many new techniques emerging recently or in the future, such as the disentangled attention mechanism from DeBERTa. Our experiments show that SAS is able to outperform ELECTRA and other state-of-the-art models in the GLUE tasks with similar or less computation cost.

Introduction

In the current landscape of Natural Language Processing (NLP), pre-training is the foundational step for achieving high performance on downstream tasks. Models like BERT established the paradigm of Masked Language Modeling (MLM), where the model learns by predicting missing words in a sentence. A significant evolution of this approach was introduced by ELECTRA, which utilized a two-network system: a Generator to replace tokens with plausible alternatives and a Discriminator to identify these replacements (Replaced Token Detection, or RTD).

While ELECTRA is highly efficient, it introduces two primary challenges. First, it requires training an auxiliary generator network that is ultimately discarded, leading to significant computational overhead. Second, it requires a delicate "balancing act": if the generator becomes too proficient too quickly, the discriminator's task becomes impossible, halting the learning process. Finding the optimal relative size and capacity between these two networks is a complex hyperparameter search problem.

Methodology: Collapsing the Generator and Discriminator

The proposed Self-Augmentation Strategy (SAS) addresses these issues by collapsing the two-network architecture into a single Transformer encoder equipped with two specialized heads: an MLM head and an RTD head. This unified model performs both generation and discrimination tasks simultaneously.

Architecture of the SAS framework illustrating the self-augmentation loop and dual-head design.

The core innovation of SAS is its temporal self-augmentation loop:

  • Epoch 1 (Cold Start): Since the model has no prior knowledge, it performs initial token replacement using a simple frequency-based unigram distribution.
  • Subsequent Epochs ($t+1$): The model utilizes the predictions generated by its own MLM head from the previous epoch to create the "fake" tokens for the current epoch's RTD task.

This approach creates an automatic curriculum: as the model improves at generating contextually relevant words (MLM), the detection task (RTD) naturally becomes more difficult. Because both tasks share the same underlying encoder, the model requires only a single forward pass, eliminating the 20-30% extra compute cost typically associated with ELECTRA's auxiliary generator.

Experimental Results

The efficacy of SAS was evaluated on the GLUE (General Language Understanding Evaluation) benchmark after pre-training on English Wikipedia and BooksCorpus (3.3 billion tokens). The results demonstrate that SAS not only simplifies the training pipeline but also achieves superior performance compared to established baselines.

Comparison of SAS variants against state-of-the-art baselines on the GLUE benchmark.

Key findings include:

  • Superior Efficiency: SAS-Small achieved a GLUE score of 81.61, outperforming the standard ELECTRA-Small (80.78) while using comparable computational resources (FLOPs).
  • Synergy with Advanced Architectures: By incorporating the disentangled attention mechanism from DeBERTa, SAS-DA-Small reached a score of 82.14. Remarkably, this outperformed "ELECTRA-Small++," a variant trained with approximately 18 times more compute.
  • Scalability: For larger configurations, SAS-Base (85.20) performed on par with ELECTRA-Base (85.46), proving the strategy's robustness across different model sizes.

Conclusion and Future Work

SAS represents a significant step toward more efficient and streamlined language model pre-training. By utilizing a model's "past self" as a contextualized data augmentor, SAS eliminates the need for auxiliary networks and the tedious tuning of generator-discriminator ratios.

Future research will focus on scaling SAS to "Large" model sizes with billions of parameters and exploring more sophisticated "warm-up" strategies to replace the initial unigram distribution. Additionally, the authors aim to further optimize the data management required for storing sampled token indices between epochs to ensure maximum training throughput.

Summary updated

Abstract

The core of self-supervised learning for pre-training language models includes pre-training task design as well as appropriate data augmentation. Most data augmentations in language model pre-training are context-independent. A seminal contextualized augmentation was recently proposed in ELECTRA and achieved state-of-the-art performance by introducing an auxiliary generation network (generator) to produce contextualized data augmentation for the training of a main discrimination network (discriminator). This design, however, introduces extra computation cost of the generator and a need to adjust the relative capability between the generator and the discriminator. In this paper, we propose a self-augmentation strategy (SAS) where a single network is utilized for both regular pre-training and contextualized data augmentation for the training in later epochs. Essentially, this strategy eliminates a separate generator and uses the single network to jointly conduct two pre-training tasks with MLM (Masked Language Modeling) and RTD (Replaced Token Detection) heads. It avoids the challenge to search for an appropriate size of the generator, which is critical to the performance as evidenced in ELECTRA and its subsequent variant models. In addition, SAS is a general strategy that can be seamlessly combined with many new techniques emerging recently or in the future, such as the disentangled attention mechanism from DeBERTa. Our experiments show that SAS is able to outperform ELECTRA and other state-of-the-art models in the GLUE tasks with similar or less computation cost.
BibTeX
@inproceedings{xu2022sas,
  author = {Yifei Xu and Jingqiao Zhang and He Ru and Liangzhu Ge and Chao Yang and Cheng Yang and Ying Wu},
  title = {{SAS: Self-Augmentation Strategy for Language Model Pre-training}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21412},
}
Figure