M2D2: A Massively Multi-domain Language Modeling Dataset

Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer

Abstract

We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar. Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups. This two-level hierarchy enables the study of relationships between domains and their effects on in- and out-of-domain performance after adaptation. We also present a number of insights into the nature of effective domain adaptation in LMs, as examples of the new types of studies M2D2 enables. To improve in-domain performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domain-specific data can lead to larger in-domain performance gains than larger amounts of weakly relevant data. We further demonstrate a trade-off between in-domain specialization and out-of-domain generalization within and across ontologies, as well as a strong correlation between out-of-domain performance and lexical overlap between domains.

Introduction

In the development of Large Language Models (LLMs), training data is frequently treated as a monolithic "pile" of text. However, the internet is a patchwork of distinct domains—ranging from quantum physics papers to culinary blogs—each characterized by unique vocabularies, stylistic conventions, and logical structures. Traditional research into domain adaptation has often focused on broad categories, such as "Medical" versus "News," which lacks the granularity required for specialized real-world applications.

M2D2 addresses this gap by providing a massive, human-curated hierarchy that allows researchers to study knowledge transfer across 145 granular subdomains. By moving beyond broad categorizations, M2D2 enables a more nuanced understanding of how models learn specific niches and how that learning affects their general capabilities.

The M2D2 Dataset

The core contribution of this work is the M2D2 dataset, which is distinguished by its scale and structural organization:

  • Scale: The dataset contains 8.5 billion tokens.
  • Granularity: It features 145 L2 (fine-grained) domains nested within 22 L1 (coarse) domains.
  • Sources: Data is sourced from Wikipedia (organized via human-curated category ontologies) and Semantic Scholar (S2ORC, organized via arXiv categories to represent academic research).
  • Hierarchy: Unlike previous datasets, M2D2’s two-level hierarchy allows for the study of relationships between closely related fields, such as "Topology" and "Number Theory" within the broader Mathematics group.

Methodology

To establish a baseline for domain adaptation, the researchers utilized a 112M parameter GPT-2 model. They evaluated three primary adaptation strategies:

  1. L1 Adaptation: Training on a broad, coarse domain (e.g., Computer Science).
  2. L2 Adaptation: Training directly on a specific, fine-grained domain (e.g., Machine Learning).
  3. L1-to-L2 Adaptation: A multi-stage hierarchical approach where the model is first adapted to the broad domain and subsequently refined on the specific subdomain.

Performance was measured using perplexity—a metric indicating how well a model predicts a sample—across various in-domain and out-of-domain tasks. Additionally, the researchers analyzed lexical indicators, such as the overlap of nouns, verbs, and entities, to determine how vocabulary similarity predicts transfer performance.

Key Insights and Results

The Advantage of Specificity and Hierarchy

A significant finding of the study is that specificity often outweighs volume. Adapting a model to a small amount of highly specific L2 data frequently results in better in-domain performance than using much larger volumes of weakly relevant L1 data.

Furthermore, the L1-to-L2 hierarchical strategy consistently yielded the best results. This suggests that models benefit from learning the general "language" of a broad field before specializing in its sub-niches. On Wikipedia data, the base GPT-2 model’s average perplexity of 26.5 was reduced to 18.8 through hierarchical adaptation. On academic data (S2ORC), the improvement was even more pronounced, with perplexity dropping from 29.2 to 15.0.

Vocabulary as a Predictor

The research identified a strong correlation between vocabulary overlap and transfer performance. Specifically, the overlap of nouns and verbs showed a Pearson correlation of approximately -0.8 with perplexity. This indicates that lexical similarity is a highly reliable predictor of how well a model adapted to one domain will perform on another.

The Specialization Trade-off

While L2 adaptation creates highly effective "experts" in specific niches, it comes at a cost. There is a clear trade-off between in-domain specialization and out-of-domain generalization. Models adapted to specific L2 domains showed significant performance degradation on unrelated tasks compared to models adapted to more general L1 domains. Additionally, the source of the data (provenance) proved critical; a model trained on Wikipedia's math section might struggle more with an academic math paper than with a Wikipedia article on history, suggesting that stylistic and structural consistency across a source is a powerful factor in transferability.

Limitations and Future Work

While M2D2 provides a robust framework for domain study, it currently treats domains as discrete units, whereas real-world text often overlaps across multiple fields. Future iterations could expand the dataset to include more informal sources, such as Reddit, to capture diverse linguistic styles. Furthermore, while these experiments were conducted on a 112M parameter model, investigating how these hierarchical benefits scale to models with hundreds of billions of parameters remains a vital area for future research.

Summary updated

Abstract

We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar. Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups. This two-level hierarchy enables the study of relationships between domains and their effects on in- and out-of-domain performance after adaptation. We also present a number of insights into the nature of effective domain adaptation in LMs, as examples of the new types of studies M2D2 enables. To improve in-domain performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domain-specific data can lead to larger in-domain performance gains than larger amounts of weakly relevant data. We further demonstrate a trade-off between in-domain specialization and out-of-domain generalization within and across ontologies, as well as a strong correlation between out-of-domain performance and lexical overlap between domains.
BibTeX
@inproceedings{reid2022m,
  author = {Machel Reid and Victor Zhong and Suchin Gururangan and Luke Zettlemoyer},
  title = {{M2D2: A Massively Multi-domain Language Modeling Dataset}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=_YmQf7ilovW},
}
Figure