Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE

Qihuang Zhong, Liang Ding, Keqin Peng, Juhua Liu, Bo Du, Li Shen, Yibing Zhan, Dacheng Tao

Abstract

Researchers at JD Explore Academy introduced Vega v1, a 1.3-billion parameter model that achieved the #1 spot on the GLUE leaderboard as of early 2022. By integrating a "bag of tricks"—including disentangled attention, a two-stage pretraining strategy, and advanced fine-tuning techniques—Vega v1 achieved a state-of-the-art average score of 91.3. Notably, it is the first model to surpass human performance on the challenging SST-2 and WNLI tasks, demonstrating that architectural and training refinements can be as impactful as raw parameter scaling.

Introduction

While Large Language Models (LLMs) have revolutionized NLP, the standard Masked Language Modeling (MLM) approach used in models like BERT is computationally inefficient, providing supervision for only a small fraction of tokens. Furthermore, adapting pretrained models to downstream tasks often encounters domain gaps and overfitting. Vega v1 addresses these challenges through a highly efficient "recipe" of architectural choices and training objectives designed to push the boundaries of Natural Language Understanding (NLU) without requiring trillion-parameter budgets.

Methodology

The Vega v1 framework relies on three primary pillars to optimize performance and efficiency:

1. Architecture: Disentangled Attention

Following the DeBERTa design, Vega v1 replaces vanilla self-attention with disentangled attention. Words are represented by two separate vectors—one for content and one for relative position—allowing the model to better capture the relationships between tokens based on their spatial context within a sentence.

2. Two-Stage Efficient Pretraining

To improve upon the sample inefficiency of MLM, the authors employed a two-stage pretraining process:

  • Phase 1 (Denoising): Utilizing Replaced Token Detection (RTD), the model performs a ternary classification (original, shuffled, or replaced) for every token, providing a much denser supervision signal.
  • Phase 2 (Contrastive Augmented): A contrastive learning objective is introduced to ensure that a sentence and its noisy version share similar embeddings. This addresses the "anisotropy" problem where embeddings cluster too narrowly, improving semantic representation.
The two-stage pretraining objectives: (a) denoising via Replaced Token Detection and (b) contrastive learning for sentence representations.

3. Advanced Adaptation Techniques

During fine-tuning, the model employs Transductive Fine-tuning (using unlabeled test data), Self-calibrated Fine-tuning (using model-labeled external data for calibration), and Adversarial Fine-tuning (adding embedding perturbations) to improve robustness and prevent overfitting on small downstream datasets.

Experimental Results

Vega v1 achieved an average GLUE score of 91.3, outperforming significantly larger models like T5 and ERNIE. It set new records on four of the nine GLUE tasks: SST-2, MRPC, QQP, and WNLI.

Radar chart showing Vega v1 performance across GLUE tasks compared to human baselines and official SOTA.

Key Insights

  • Human-Level Reasoning: Vega v1's success on WNLI (97.9% vs. human 95.9%) is particularly significant, as the task requires deep coreference resolution and logical reasoning.
  • Efficiency Gains: The results reinforce that RTD is a more efficient objective than MLM for learning basic linguistics, enabling SOTA performance with a "medium-sized" 1.3B parameter count.
  • Strategic Training: By introducing the computationally expensive contrastive objective only in the second phase, the authors balanced performance gains with training costs.

Conclusion

Vega v1 demonstrates that a sophisticated "bag of tricks" can lead to superior NLU performance. While the authors note that the RTD objective may face stability issues when scaling beyond 5 billion parameters, the current results provide a powerful blueprint for efficient model pretraining and adaptation. Future work will focus on stabilizing these efficient objectives for even larger-scale architectures.

Summary updated

Abstract

This technical report briefly describes our JDExplore d-team's submission Vega v1 on the General Language Understanding Evaluation (GLUE) leaderboard, where GLUE is a collection of nine natural language understanding tasks, including question answering, linguistic acceptability, sentiment analysis, text similarity, paraphrase detection, and natural language inference. [Method] We investigate several effective strategies and choose their best combination setting as the training recipes. As for model structure, we employ the vanilla Transformer with disentangled attention as the basic block encoder. For self-supervised training, we employ the representative denoising objective (i.e., replaced token detection) in phase 1 and combine the contrastive objective (i.e., sentence embedding contrastive learning) with it in phase 2. During fine-tuning, several advanced techniques such as transductive fine-tuning, self-calibrated fine-tuning, and adversarial fine-tuning are adopted. [Results] According to our submission record (Jan. 2022), with our optimized pretraining and fine-tuning strategies, our 1.3 billion model sets new state-of-the-art on 4/9 tasks, achieving the best average score of 91.3. Encouragingly, our Vega v1 is the first to exceed powerful human performance on the two challenging tasks, i.e., SST-2 and WNLI. We believe our empirically successful recipe with a bag of tricks could shed new light on developing efficient discriminative large language models.
BibTeX
@misc{zhong2023bag,
  author = {Qihuang Zhong and Liang Ding and Keqin Peng and Juhua Liu and Bo Du and Li Shen and Yibing Zhan and Dacheng Tao},
  title = {{Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE}},
  year = {2023},
  eprint = {2302.09268},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.09268},
}
Figure