M2D2: A Massively Multi-Domain Language Modeling Dataset

Machel Reid, Victor Zhong, Suchin Gururangan, Luke Zettlemoyer

Introduction

Most research in domain adaptation for Large Language Models (LLMs) treats "domains" as broad, monolithic categories such as "Legal," "Medical," or "Scientific." However, human knowledge is naturally a granular hierarchy; for instance, "Machine Learning" is a subset of "Computer Science," which in turn is a subset of "Technology." Existing benchmarks often lack the scale and structural depth to study how LMs transfer knowledge between these nested layers. M2D2 was created to provide a massive, structured testbed to answer these questions, exploring whether it is better to train on a large volume of vaguely related data or a small sliver of perfectly related data.

The M2D2 Dataset

The M2D2 dataset features 8.5 billion tokens across 145 fine-grained (L2) domains nested within 22 coarse (L1) groups. The data is sourced from two primary repositories:

  • Wikipedia: Organized using the site's internal category ontologies.
  • Semantic Scholar (S2ORC): Organized using arXiv categories.

Unlike previous datasets that might only include 4–20 broad domains, M2D2’s 145 domains allow for the rigorous study of "data-scarce" niches recognized by human curators.

Adaptation Methodology

The authors established a baseline using a 112M parameter GPT-2 model and explored three primary adaptation strategies to navigate the hierarchy:

  1. L1 Adaptation: Fine-tuning on a broad category (e.g., Computer Science).
  2. L2 Adaptation: Fine-tuning directly on a specific subdomain (e.g., Machine Learning).
  3. L1-to-L2 Adaptation: A multi-stage approach where the model is first adapted to the broad L1 category and then further refined on the specific L2 subdomain.
Hierarchical adaptation strategies: L1, L2, and L1-to-L2.

To understand the mechanics of transfer, they also performed lexical analysis using Part-of-Speech (POS) tagging and Named Entity Recognition (NER) to calculate vocabulary overlap between domains.

Results and Discussion

The "Coarse-to-Fine" Advantage

The most effective way to adapt a model is to follow the hierarchy. Training on a broad category before a specific one (L1-to-L2) consistently outperformed direct L2 adaptation. On Wikipedia data, the base GPT-2 had a perplexity of 26.5, which dropped to 18.8 using L1-to-L2 adaptation. The effect was even more pronounced on academic data (S2ORC), as shown in the table below.

Perplexity results for different adaptation methods across S2ORC domains.

The Specialization Tax and Provenance

While adaptation significantly lowers in-domain perplexity, there is a clear trade-off: as a model becomes a specialist in a fine-grained domain, its generalist performance (out-of-domain) degrades significantly. Surprisingly, the source of the data (Wikipedia vs. S2ORC) is often a better predictor of transferability than the topic. For example, a model trained on Wikipedia "Mathematics" transfers better to Wikipedia "Art" than it does to S2ORC "Mathematics."

Vocabulary as a Proxy

Simple lexical overlap—specifically nouns and entities—is a remarkably strong predictor of how well a model will transfer between two domains. The researchers found a strong negative correlation (Pearson r ≈ -0.8) between vocabulary overlap and out-of-domain perplexity, suggesting that shared entities are the primary "bridge" for knowledge transfer.

Conclusion and Future Work

M2D2 demonstrates that specificity trumps volume: adapting to a small amount of highly specific data leads to larger in-domain gains than using a much larger volume of weakly relevant data. However, the resulting "specialization tax" remains a challenge. Future research should investigate ensembling or adapter-based methods to retain general capabilities while specializing, and explore whether these hierarchical benefits scale linearly with billion-parameter models.

Summary updated

Abstract

We present M2D2, a fine-grained, massively multi-domain corpus for studying domain adaptation in language models (LMs). M2D2 consists of 8.5B tokens and spans 145 domains extracted from Wikipedia and Semantic Scholar. Using ontologies derived from Wikipedia and ArXiv categories, we organize the domains in each data source into 22 groups. This two-level hierarchy enables the study of relationships between domains and their effects on in- and out-of-domain performance after adaptation. We also present a number of insights into the nature of effective domain adaptation in LMs, as examples of the new types of studies M2D2 enables. To improve in-domain performance, we show the benefits of adapting the LM along a domain hierarchy; adapting to smaller amounts of fine-grained domain-specific data can lead to larger in-domain performance gains than larger amounts of weakly relevant data. We further demonstrate a trade-off between in-domain specialization and out-of-domain generalization within and across ontologies, as well as a strong correlation between out-of-domain performance and lexical overlap between domains.
BibTeX
@inproceedings{reid2022m,
  author = {Machel Reid and Victor Zhong and Suchin Gururangan and Luke Zettlemoyer},
  title = {{M2D2: A Massively Multi-Domain Language Modeling Dataset}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=_YmQf7ilovW},
}
Figure