Connecting Pre-trained Language Model and Downstream Task via Properties of Representation

Chenwei Wu, Holden Lee, Rong GeNeurIPS 2023 · OpenReview

Introduction

Large-scale pre-trained language models (LLMs) have demonstrated remarkable empirical success, acting as versatile foundations for a wide array of downstream tasks such as sentiment analysis and reading comprehension. Despite this success, a theoretical understanding of why pre-training performance translates so effectively to downstream tasks remains elusive. This paper bridges that gap by identifying a hidden structural property in LLM representations—the "anchor vector"—which provides a mathematical guarantee that low pre-training loss will yield low error on downstream classification tasks.

The Theoretical Challenge

Traditional learning theory relies on the assumption that training and test data are drawn from the same distribution. In the context of LLMs, this assumption is violated: models are pre-trained on massive, diverse web corpora (language modeling) but evaluated on specific, narrow domains (e.g., news classification).

Furthermore, the "softmax" layer used in these models introduces a mathematical complication: it is shift-invariant. This means a constant can be added to all internal scores (logits) without altering the final output probabilities. Because softmax only evaluates the relative difference between scores, a model could theoretically achieve near-perfect pre-training loss while maintaining unstable or arbitrary internal representations. Without additional structural constraints, pre-training alone does not strictly guarantee downstream success.

Proposed Methodology

To connect pre-training and downstream tasks, the authors analyze a log-linear model where words are predicted from context via a hidden representation and a softmax layer. They introduce two critical components to bridge the theoretical gap:

  1. The Anchor Vector Hypothesis: The authors propose the existence of a fixed "anchor vector" ($v_0$) within the representation space. This vector serves to estimate the "bulk partition function"—essentially the background noise generated by the thousands of vocabulary words that are irrelevant to a specific downstream task.
  2. Refined Task Modeling: Downstream tasks are assumed to be insensitive to words with infinitesimally small probabilities. The authors model this using a ReLU-based function that ignores logits below a certain threshold, preventing the model from being distracted by the noise of rare words. This aligns with the idea of "k-sparsity," where a task depends only on a small subset of important words.

Empirical Validation

The researchers validated the anchor vector hypothesis using various configurations of GPT-2 (Base, Medium, Large, XL) and OPT (125M to 1.3B parameters) on the WikiText-2 corpus.

Distributions of the original and bulk partition functions, illustrating significant variance across samples.

Empirical observations revealed that the "bulk partition function" (the sum of exponentials of non-top-k words) varies significantly across different text samples. However, the proposed anchor vector is remarkably capable of tracking these variations.

Mean square approximation error of the anchor vector across different models and values of k.

By measuring the Mean Squared Error (MSE), the authors demonstrated that the anchor vector can approximate the log bulk partition function with high precision. The approximation error was found to be several orders of magnitude smaller than the actual values, confirming that the anchor vector is a real and measurable phenomenon in modern LLMs.

Key Findings

  • Theoretical Guarantee: The paper mathematically proves that if an anchor vector exists and the downstream task is well-behaved (i.e., it has a margin and ignores negligible probabilities), the downstream classification error ($\\epsilon_{CLS}$) is bounded by the pre-training KL divergence loss ($\\epsilon_{KL}$) scaled by constant factors. As language modeling loss decreases, downstream performance is mathematically guaranteed to improve.
  • Non-Uniformity of Embeddings: The analysis shows that word embeddings in modern LLMs are not uniformly distributed on a hypersphere. This non-uniformity makes the existence of the anchor vector a highly non-trivial property of how these models organize semantic information.

Limitations and Future Directions

While this work provides a crucial theoretical stepping stone, it has limitations. The theory currently relies on a sparse word assumption, which may not fully capture complex, multi-step reasoning tasks where the entire context is equally critical. Additionally, the analysis focuses exclusively on the representations at the final layer, leaving the dynamics of intermediate Transformer layers and the effects of full-model fine-tuning unexplored.

Finally, while the paper proves that the anchor vector exists, it leaves open the question of why the training process creates it. Whether this property emerges from the Transformer architecture, the optimization algorithm, or the statistical nature of the training data remains a compelling question for future research.

Summary updated

Abstract

Recently, researchers have found that representations learned by large-scale pre-trained language models are useful in various downstream tasks. However, there is little theoretical understanding of how pre-training performance is related to downstream task performance. In this paper, we analyze how this performance transfer depends on the properties of the downstream task and the structure of the representations. We consider a log-linear model where a word can be predicted from its context through a network having softmax as its last layer. We show that even if the downstream task is highly structured and depends on a simple function of the hidden representation, there are still cases when a low pre-training loss cannot guarantee good performance on the downstream task. On the other hand, we propose and empirically validate the existence of an ``anchor vector'' in the representation space, and show that this assumption, together with properties of the downstream task, guarantees performance transfer.
BibTeX
@inproceedings{wu2023connecting,
  author = {Chenwei Wu and Holden Lee and Rong Ge},
  title = {{Connecting Pre-trained Language Model and Downstream Task via Properties of Representation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=YLOJ4aKAka},
}
Figure