On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets

David Cheng-Han Chiang, Hung-yi Lee

Abstract

Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.

Introduction

Masked Language Modeling (MLM) has become the standard for pre-training powerful language models. While these models consistently outperform those trained from scratch on downstream tasks, the underlying reasons for this success remain partially obscured. Traditional views suggest that pre-training captures semantic dependencies or complex linguistic structures. However, isolating these factors in natural language is challenging due to the inherent entanglement of semantics and syntax.

This study investigates whether semantics are strictly necessary for transferability. By pre-training models on artificial datasets—sequences of integers governed by specific structural rules—we can isolate the impact of data characteristics like token distribution and dependency length on downstream performance.

Methodology

The core approach involves a three-stage pipeline: pre-training on an artificial "Language 1" (L1), fine-tuning on English downstream tasks (GLUE), and testing. During pre-training, the model (RoBERTa-medium) learns to predict masked integer tokens. For fine-tuning, these integer embeddings are mapped to English sub-words. This ensures that no semantic knowledge is transferred; only the structural processing capabilities of the transformer weights are tested.

The three-stage experimental pipeline: MLM pre-training on artificial data, GLUE fine-tuning, and testing.

Artificial Dataset Design

To isolate specific traits, the authors designed several artificial L1 datasets:

  1. Distributional Baselines: Tokens are sampled to match Uniform, Uni-gram, or Bi-gram distributions of English.
  2. Explicit Dependencies: These datasets use structural rules where tokens are paired. "Flat Parentheses" involve pairs separated by a fixed distance, while "Nesting Parentheses" use a stack-based hierarchical structure.
  3. Implicit Dependencies: "Shuffle-N" datasets are created by shuffling blocks of $N$ consecutive integers, requiring the model to learn local context to resolve masks.
Structural rules for artificial datasets, including Flat Parentheses, Nesting Parentheses, and Shuffle-4.

Key Results and Insights

The experiments reveal that structural dependencies are far more important than simple token statistics. While matching uni-gram or bi-gram distributions provided negligible gains (+1.3 to +2.2 GLUE points over scratch), datasets with long-range dependencies showed dramatic improvements.

  • Dependency Length Matters: The Flat-128 dataset, featuring dependencies spanning 128 tokens, achieved a +14.5 GLUE score improvement, nearly matching a human-language baseline (Kannada).
  • Hierarchy vs. Distance: Interestingly, the hierarchical "Nesting Parentheses" did not significantly outperform "Flat Parentheses," suggesting that the ability to model long-range dependencies is more critical for transfer than learning specific hierarchical syntax.
  • Vulnerability to Spurious Correlations: Despite high GLUE scores, models pre-trained on artificial data are fragile. On adversarial tests like QNLI-Adv, accuracy plummeted to ~1%, indicating that these models rely heavily on lexical overlap heuristics rather than robust reasoning.

Conclusion

This research demonstrates that pre-trained LMs gain significant transferability by learning to model token dependencies, even in the absence of natural language semantics. The results suggest that the "magic" of pre-training lies largely in the model's acquired ability to process structured sequences. Future work will explore these findings in the context of generative (causal) models and sequence-to-sequence architectures.

Summary updated

Abstract

Pre-training language models (LMs) on large-scale unlabeled text data makes the model much easier to achieve exceptional downstream performance than their counterparts directly trained on the downstream tasks. In this work, we study what specific traits in the pre-training data, other than the semantics, make a pre-trained LM superior to their counterparts trained from scratch on downstream tasks. We propose to use artificially constructed datasets as the pre-training data to exclude the effect of semantics, and further control what characteristics the pre-training corpora have. By fine-tuning the pre-trained models on GLUE benchmark, we can learn how beneficial it is to transfer the knowledge from the model trained on the dataset possessing that specific trait. We define and discuss three different characteristics in the artificial dataset: 1) matching the token's uni-gram or bi-gram distribution between pre-training and downstream fine-tuning, 2) the presence of the explicit dependencies among the tokens in a sequence, 3) the length of the implicit dependencies among the tokens in a sequence. Our experiments show that the explicit dependencies in the sequences of the pre-training data are critical to the downstream performance. Our results also reveal that models achieve better downstream performance when pre-trained on a dataset with a longer range of implicit dependencies. Based on our analysis, we find that models pre-trained with artificial datasets are prone to learn spurious correlation in downstream tasks. Our work reveals that even if the LMs are not pre-trained on natural language, they still gain transferability on certain human language downstream tasks once the LMs learn to model the token dependencies in the sequences. This result helps us understand the exceptional transferability of pre-trained LMs.
BibTeX
@inproceedings{chiang2022transferability,
  author = {David Cheng-Han Chiang and Hung-yi Lee},
  title = {{On the Transferability of Pre-trained Language Models: A Study from Artificial Datasets}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=sq2Gb59woq},
}
Figure