Pre-trained Models for Representation Learning

Yankai Lin, Ning Ding, Zhiyuan Liu, Maosong SunOpenalex Unknown 2023 · DOI · OpenAlex

Abstract

Pre-training-fine-tuning has recently become a new paradigm in natural language processing, learning better representations of words, sentences, and documents in a self-supervised manner. Pre-trained models not only unify semantic representations of multiple tasks, multiple languages, and multiple modalities but also emerge high-level capabilities approaching human beings. In this chapter, we introduce pre-trained models for representation learning, from pre-training tasks to adaptation approaches for specific tasks. After that, we discuss several advanced topics toward better pre-trained representations, including better model architecture, multilingual, multi-task, efficient representations, and chain-of-thought reasoning.

Introduction: The Shift Toward Unified Representations

Historically, machine learning for Natural Language Processing (NLP) was fragmented. Statistical models required manual feature engineering, while early deep learning models necessitated massive labeled datasets for every individual task. This approach was inherently inefficient and failed to capture the general "common sense" of language.

The emergence of the "Pre-training-Fine-tuning" paradigm addresses these challenges by creating unified representations. Instead of task-specific architectures, a single "foundation model" learns from vast quantities of unlabeled data to handle translation, summarization, and reasoning simultaneously. This shift represents a move toward a universal semantic space where multiple modalities and languages can coexist.

The shift from task-specific fine-tuning to unified pre-trained representations.

Methodology: Pre-training and Adaptation

The methodology of modern NLP is built upon three main pillars: pre-training objectives, model adaptation, and advanced architectures.

Pre-training Objectives

Models are trained using self-supervised objectives to learn contextualized embeddings:

  • Word-level: Techniques like Masked Language Modeling (MLM) (e.g., BERT), Causal Language Modeling (CLM) (e.g., GPT), and Replaced Language Modeling (e.g., ELECTRA) allow models to understand context.
  • Sentence-level: Relationships between text blocks are learned via Next Sentence Prediction (NSP) and Sentence Contrastive Learning (SCL).

Model Adaptation

As models grow in size, traditional fine-tuning (updating all parameters) becomes computationally prohibitive. Modern adaptation techniques include:

  • Delta Tuning: Methods like Adapters, Prefix Tuning, and LoRA (Low-Rank Adaptation) update less than 1% of parameters while maintaining performance.
  • Prompt Learning: Bridging the gap between pre-training and downstream use by reformulating tasks as "fill-in-the-blank" problems.

Scaling Laws and Emergent Capabilities

The progression of Pre-trained Models (PTMs) has been characterized by an exponential increase in parameter count and data volume. Tracking the evolution from BERT (340M parameters) to the Switch-Transformer (1.6T parameters) reveals that we have not yet reached the ceiling of performance gains from scaling.

The evolution of pre-trained models across modalities and parameter scales (2016-2022).

A critical finding in recent research is the existence of emergent properties. For instance, complex Chain-of-Thought (CoT) reasoning—the ability of a model to "show its work"—only reliably appears once models exceed a threshold of approximately 100 billion parameters. Furthermore, GPT-3 demonstrated that "in-context learning" (few-shot performance) can often outperform smaller models that have been fully fine-tuned on thousands of examples.

Key Insights and Future Directions

Research into these models has yielded several profound insights:

  • Intrinsic Sparsity: Vanilla Transformers may be a special case of Mixture of Experts (MoE). Sparsity appears to be an emergent property, allowing models to increase capacity without a linear increase in computational cost.
  • Low-Dimensional Adaptation: The optimization space for adapting a model to a new task is intrinsically low-dimensional, explaining why parameter-efficient methods like LoRA are so effective.

Despite these successes, challenges remain. The "black box" nature of reasoning, data imbalances in low-resource languages, and high inference costs are ongoing hurdles. Future work is increasingly focused on Human Alignment through Reinforcement Learning from Human Feedback (RLHF) to ensure these intelligent agents behave according to human values and instructions.

Summary updated

Abstract

Abstract Pre-training-fine-tuning has recently become a new paradigm in natural language processing, learning better representations of words, sentences, and documents in a self-supervised manner. Pre-trained models not only unify semantic representations of multiple tasks, multiple languages, and multiple modalities but also emerge high-level capabilities approaching human beings. In this chapter, we introduce pre-trained models for representation learning, from pre-training tasks to adaptation approaches for specific tasks. After that, we discuss several advanced topics toward better pre-trained representations, including better model architecture, multilingual, multi-task, efficient representations, and chain-of-thought reasoning.
BibTeX
@inproceedings{lin2023pre,
  author = {Yankai Lin and Ning Ding and Zhiyuan Liu and Maosong Sun},
  title = {{Pre-trained Models for Representation Learning}},
  booktitle = {Openalex Unknown},
  year = {2023},
  doi = {10.1007/978-981-99-1600-9\_5},
}
Figure