TL;DR
This paper investigates how the choice of pre-training corpus affects the quality of learned transformer representations. They find that smaller, specialized corpora can be as effective as large, general ones, and that domain-adaptive pre-training's success hinges on the similarity between the pre-training and target task domains.
What Problem Does This Paper Address?
Large language models (LLMs) like BERT and GPT-2 are typically pre-trained on massive, general-domain text corpora. This pre-training imbues them with powerful representations that can be fine-tuned for various downstream tasks. However, for specific applications (e.g., a toxicity filter for a niche forum), using a general corpus might not be optimal. The question is: how does the choice of pre-training data—specifically, using a smaller, domain-specific corpus versus a large, general one, or a combination—impact the quality of the learned representations? This paper aims to empirically analyze this, focusing on representation quality after pre-training, without subsequent supervised fine-tuning.
Methods
The authors employ three main metrics to evaluate representation quality:
- Low-Annotation Probing (ALC%): This uses weak classifiers trained on very few labeled examples to assess how well the representations capture underlying task structures. They report the area under the learning curve (ALC) for probes trained on 100 to 1000 samples.
- Task Hierarchical Alignment Score (THAS%): This metric quantifies how well the representation space aligns with hierarchical clustering structures derived from task labels.
- Data-Dependent Complexity (DDC%): This measures the compatibility between a representation and a binary classification task by analyzing how well label vectors can be reconstructed from projections onto the representation's principal components.
They compare three pre-training scenarios:
- In-domain (ID): Training a model from scratch on a domain-specific corpus (S).
- General Domain (GD): Using a standard, large, generic corpus (G), like BERT on BookCorpus/Wikipedia or GPT-2 on WebText.
- Domain-Adaptive (DA): Continuing pre-training on a domain-specific corpus (S) after initial pre-training on a general corpus (G).
Datasets
Pre-training Corpora: * General Corpora (G): BookCorpus and English Wikipedia (for BERT), WebText (for GPT-2). * Domain-Specific Corpora (S): Unlabeled text from various classification datasets, including IMDb, Sentiment 140, Wiki Toxic, Civil Comments, TREC, AG-News, and Ohsumed.
Evaluation Tasks: The evaluation tasks correspond to the original classification tasks of the pre-training corpora. Additionally, they created two new multi-class toxicity datasets: Wiki Toxic5 and Civil Comments6, derived from existing toxicity datasets by focusing on specific subtype annotations.
Key Insights and Ideas
- Specialized Data Can Shine: Pre-training on a smaller, specialized corpus (S) can yield representations comparable in quality to those from a much larger, general corpus (G), provided S is sufficiently large and relevant to the target task. This suggests data relevance can be more important than sheer size.
- Domain-Adaptive Trade-offs: Domain-adaptive pre-training (G+S) can sometimes improve performance over GD alone, but it can also lead to degraded performance. This outcome is not guaranteed and depends heavily on the relationship between the domains.
- Domain Similarity is Key: The success of domain-adaptive pre-training is strongly correlated with the distributional similarity between the pre-training domain (S) and the target task domain (Stask). This is a crucial finding for understanding cross-domain generalization.
- BERT vs. GPT-2: While the general trends are similar between BERT and GPT-2, GPT-2's in-domain representations often lag behind its general-domain counterpart. This might be due to the extremely strong baseline established by GPT-2's extensive WebText pre-training.
- Clustering Quality vs. Task Performance: Interestingly, the paper finds that improvements in representation quality (measured by task alignment and probing) are not always correlated with improvements in embedding space clustering quality (measured by DBI scores). Pre-training can alter cluster structures in ways that don't necessarily lead to more compact or separable clusters.
Key Results
- In-domain Pre-training: For domain-matched cases, in-domain pre-training on specialized corpora achieved performance comparable to general-domain pre-training, especially with sufficient training steps. For example, on the IMDb dataset, in-domain BERT achieved similar task alignment scores to general-domain BERT.
- Domain-Adaptive Pre-training: This strategy yielded the most substantial improvements in representation quality when the pre-training domain was closely related to the target task. For instance, using domain-adaptive pre-training on the Wiki Toxic corpus for toxicity-related tasks showed significant gains.
- Distributional Similarity Correlation: The paper quantifies the correlation between domain similarity metrics (n-gram coverage and expected L1-accuracy) and representation quality improvements. They found strong positive correlations, particularly for the L1-accuracy metric, supporting their hypothesis.
- Corpus Size Matters (to an extent): While relevance is key, extremely small corpora (like TREC or Ohsumed) struggled to produce high-quality representations, even in-domain, suggesting corpus size is still a limiting factor.
Limitations and Future Work
Limitations: * Architecture Scope: The study primarily focuses on BERT and GPT-2 architectures. While findings are likely generalizable, a broader comparison across different transformer architectures could provide more robust conclusions. * Computational Resources: Pre-training very large LMs is computationally intensive, limiting the scale of experiments. * Fine-tuning Ignored: The work focuses solely on pre-training effects, not on the impact of subsequent supervised fine-tuning, which is a common practice. * Task Type Restriction: The DDC metric is restricted to binary classification tasks, limiting its application to other task types. The evaluation is also largely confined to classification tasks. * Clustering Metric Limitations: The DBI score analysis suggests that while representations improve for specific tasks, their overall clustering quality might not improve or could even degrade, indicating a complex relationship between task-specific performance and global embedding space structure.
Future Work: * Investigating the impact of fine-tuning on representation quality. * Expanding the experimental setup to include a wider range of tasks beyond classification. * Exploring other transformer architectures. * Developing more appropriate parallel metrics to capture representation changes across a broader range of task types.
Summary updated