Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work

Cheng-Han Chiang, Yung-Sung Chuang, Hung-yi Lee2022 · DOI

Abstract

Since the emergence of BERT and ELMo, Pre-trained Language Models (PLMs) have fundamentally reshaped the landscape of Natural Language Processing (NLP). This paper provides a comprehensive roadmap of the "post-BERT" era (2020–2022), synthesizing research to address two core questions: why these models are so effective and how their training and adaptation can be optimized for efficiency. By categorizing hundreds of studies into a structured framework, the authors bridge the gap between empirical success and theoretical understanding while highlighting the shift toward resource-efficient AI.

Introduction

The introduction of Transformer-based PLMs marked a paradigm shift in NLP, moving from task-specific architectures to a "pre-train then fine-tune" workflow. However, this success brought significant challenges. Initially, PLMs were treated as "black boxes," with little understanding of the linguistic or factual knowledge encoded within their billions of parameters. Furthermore, the escalating computational costs of training and fine-tuning these models created a "resource barrier," limiting cutting-edge research to organizations with massive computing infrastructure. This paper addresses these challenges by surveying the literature on model interpretability and efficiency.

A Taxonomy of PLM Research

The authors organize the vast landscape of recent PLM research into two primary parts: understanding the underlying mechanisms ("Why") and improving the practical application ("How").

Taxonomy of PLM research categorized by theoretical/empirical explanations and methodological innovations.

Part I: Why Do They Work?

Understanding the efficacy of PLMs involves two distinct approaches:

  1. Empirical Explanations: Researchers use probing tasks to determine if models capture syntax, word order, or world facts. These studies reveal that while PLMs are proficient at capturing linguistic structure, their "understanding" of facts is often distinct from human-like reasoning.
  2. Theoretical Explanations: This branch employs mathematical frameworks to bound generalization errors. By analyzing the mechanics of transfer learning, theorists aim to explain why pre-training on a general corpus significantly improves performance on specific downstream tasks.

Part II: How Do They Work?

The "How" focuses on making PLMs more accessible and effective through three main avenues:

  • Pre-training Innovations: To combat the "resource-hungry" nature of initial models, new techniques like token dropping and implementation optimizations have emerged. Additionally, contrastive learning—a technique popularized in computer vision—has been adapted to improve the quality of sentence embeddings.
  • Fine-tuning Paradigms:
    • Parameter-Efficient Fine-tuning (PEFT): Methods such as Adapters, Prefix-tuning, and Low-Rank Adaptation (LoRA) allow models to be adapted to new tasks by updating only a tiny fraction (often <1%) of their parameters.
  • Data-Efficient Fine-tuning: This includes semi-supervised, few-shot, and zero-shot learning strategies, often leveraged through "prompting" techniques that frame tasks as natural language queries.
  • Cross-task Transfer: This involves using intermediate tasks or multi-task learning to provide a better starting point for target task fine-tuning.

Several critical trends have emerged from the 2020–2022 research cycle:

  • The Efficiency Revolution: A major realization in the field is that full fine-tuning is often unnecessary. PEFT methods can match the performance of full fine-tuning while drastically reducing storage and compute requirements.
  • Implicit Knowledge Bases: PLMs act as implicit knowledge bases, storing vast amounts of factual information. However, the way this information is retrieved and utilized remains a subject of intense study.
  • The Power of Prompting: The emergence of zero-shot and few-shot capabilities is not solely a function of model size but also of how tasks are presented to the model. Prompt engineering has become a vital tool for unlocking the latent capabilities of PLMs.
  • Cross-Domain Migration: Techniques like contrastive learning demonstrate the benefit of cross-pollination between computer vision and NLP, leading to more robust representations.

Conclusion and Future Directions

While the 2020–2022 period saw immense progress in democratizing PLM research through efficiency, several hurdles remain. The "Why" behind PLMs is still an evolving field, with a complete mathematical theory yet to be established. Furthermore, the rapid rise of Large Language Models (LLMs) and instruction tuning suggests that the field is moving toward even larger scales, potentially re-introducing the resource barriers that efficiency research sought to lower. For practitioners, the challenge remains deploying these powerful models in realistic, low-resource settings.

Summary updated

BibTeX
@misc{chiang2022recent,
  author = {Cheng-Han Chiang and Yung-Sung Chuang and Hung-yi Lee},
  title = {{Recent Advances in Pre-trained Language Models: Why Do They Work and How Do They Work}},
  year = {2022},
  doi = {10.18653/v1/2022.aacl-tutorials.2},
}
Figure