On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

David Cheng-Han Chiang, Hung-Yi Lee

Abstract

Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.

Introduction

The success of Masked Language Modeling (MLM) is often attributed to the model's ability to learn the "semantics" or "meaning" of a language. However, it remains unclear which specific traits of the pre-training data actually facilitate this transfer of knowledge. Is it the vocabulary? The statistical distribution of words? Or the underlying grammatical structure? This paper aims to disentangle these factors by removing semantics entirely, investigating whether pre-training on artificial languages governed by specific structural rules can still benefit downstream English tasks.

Methodology

The researchers employed a controlled experimental framework using a RoBERTa-medium architecture. The process followed three distinct stages to isolate the effects of pre-training data characteristics.

The three-stage experimental framework: MLM pre-training on artificial data (Stage 1), fine-tuning on GLUE (Stage 2), and testing on English tasks (Stage 3).
  1. Pre-training: The model is trained using MLM on an artificial "Language 1" (L1) consisting of integer sequences.
  2. Fine-tuning: The pre-trained model is then fine-tuned on English downstream tasks (GLUE benchmark). Crucially, because the L1 vocabulary (integers) has no overlap with English, the model cannot transfer any semantic knowledge.
  3. Testing: The model is evaluated on standard English test sets.

Artificial Dataset Design

To isolate what matters, the authors designed several types of artificial datasets to test different statistical and structural properties:

Visual representation of artificial dataset structures: (a) Flat Parentheses, (b) Nesting Parentheses, and (c) Shuffle-4.
  • Distributional: Datasets matching the frequency of single words (uni-grams) or pairs (bi-grams) in English.
  • Explicit Dependencies: "Parentheses" datasets where tokens appear in pairs (like open/close brackets), forcing the model to learn long-distance relationships.
  • Implicit Dependencies: "Shuffle-N" datasets where blocks of $N$ integers are shuffled, requiring the model to learn local context dependencies.

Key Results and Insights

Structure Trumps Semantics

Surprisingly, pre-training on artificial data with zero natural language meaning can make a model perform significantly better than training from scratch. Models pre-trained on Flat-128 (long explicit dependencies) showed a +14.5 average point improvement over training from scratch. For comparison, pre-training on the human language Kannada yielded a +14.7 improvement, suggesting that structural learning accounts for a massive portion of pre-training benefits.

Dependencies are the "Secret Sauce"

Simply matching word frequencies (uni-grams/bi-grams) isn't enough. The model gains the most transferability when it is forced to learn dependencies—rules that dictate how one token relates to another in a sequence. Furthermore, the "range" of the dependency is critical; models pre-trained on data with longer-range dependencies consistently achieved higher scores on downstream English tasks.

The Spurious Correlation Trap

While artificial pre-training helps with accuracy, it makes models "brittle." These models are much more likely to rely on surface-level patterns (spurious correlations) than models pre-trained on real English. On the QNLI-Adv task, the English-trained model maintained 64.8% accuracy. In contrast, models pre-trained on Nesting Parentheses or Shuffle-4 crashed to ~1% accuracy, performing worse than random guessing. This suggests that while artificial data teaches the model how to process sequences, it doesn't teach it the robustness inherent in natural language.

Conclusion

This study reveals that even without natural language semantics, pre-trained LMs gain transferability once they learn to model token dependencies. The explicit dependencies in pre-training data are critical to downstream performance, and longer dependency ranges further enhance this effect. While these models are prone to learning spurious correlations, the results provide a new perspective on why pre-trained models are so effective: they learn the fundamental structural patterns of sequence modeling that are universal across languages and even artificial systems.

Summary updated

Abstract

Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.
BibTeX
@inproceedings{chiang2022transferability,
  author = {David Cheng-Han Chiang and Hung-Yi Lee},
  title = {{On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=sq2Gb59woq},
}
Figure