DEMix Layers: Disentangling Domains for Modular Language Modeling

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer

Abstract

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.

Introduction

Standard Large Language Models (LLMs) are typically "dense," meaning every parameter is updated for every piece of data during training. This approach assumes data homogeneity, which contradicts the diverse nature of human language. Dense training leads to several critical flaws: computational inefficiency as models scale, "catastrophic forgetting" during domain adaptation, a lack of control for "unlearning" specific data subsets, and poor generalization on the "long tail" of less prevalent topics.

To address these issues, we propose DEMix (Domain Expert Mixture) layers, a modular approach that replaces standard feedforward networks with a collection of domain-specific "experts." By explicitly conditioning on text domains, DEMix models achieve better generalization and the unique ability to modify knowledge post-training.

Methods: The DEMix Layer

The core innovation is the DEMix layer. In a standard Transformer, the feedforward network (FFN) is a single block. DEMix replaces this with $n$ parallel FFNs, each acting as an expert for a specific domain (e.g., legal, medical, or news).

The DEMix layer architecture supports specialized training and flexible inference operations like mixing, adding, or removing experts.

Routing and Training

Unlike traditional Mixture-of-Experts (MoE) models that learn token-level routing, DEMix utilizes metadata-based routing. During training, the model uses document provenance to route the entire sequence to the corresponding domain expert. This ensures that experts truly specialize in their assigned domains and simplifies the training process.

Inference-Time Flexibility

At inference, DEMix allows for dynamic adaptation to unknown or overlapping domains. The model calculates a posterior probability $p(\text{domain} | \text{text})$ using Bayes' rule, enabling a weighted ensemble of expert outputs. This allows the model to handle heterogeneous text by mixing relevant experts.

Inference-time mixing: The model estimates domain probabilities to combine expert outputs for novel inputs, such as a COVID-19 paper.

Experimental Setup

We evaluated DEMix using a multi-domain corpus of 73.8 billion tokens.

  • Training Domains: Eight diverse sources, including news (RealNews), scientific papers (CS, Med), legal opinions, web text, Reddit, and Amazon reviews.
  • Novel Domains: Eight additional datasets (e.g., GitHub code, Tweets, Gutenberg books) were used to test out-of-distribution generalization.

Key Results and Insights

Efficiency and Specialization

DEMix reduces communication overhead in distributed training because experts only synchronize with GPUs assigned to the same domain. This leads to higher throughput as parameter counts grow. Furthermore, experts exhibit strong specialization; for instance, a "medical" expert significantly outperforms a "legal" expert on medical text, confirming successful knowledge disentanglement.

Generalization through Mixing

For novel domains, a weighted mixture of experts consistently outperforms naively selecting the single "closest" expert. For example, a paper on COVID-19 is best processed by a mixture of Medical and News experts.

Adaptation and Removability

DEMix enables adding new experts for new domains (DEMix-DAPT) while freezing the rest of the model. This achieves high performance on new data with zero performance loss on original training domains, avoiding catastrophic forgetting. Conversely, experts can be removed (e.g., disabling a Reddit expert) to simulate a model never exposed to that domain, providing a mechanism for data control without retraining.

Conclusion

DEMix layers demonstrate the benefits of explicitly conditioning on textual domains. By modularizing the FFN blocks, we achieve a model that is more efficient, adaptable, and controllable. Future work will explore unsupervised methods for domain discovery to reduce dependency on human-provided metadata and scale to a larger number of granular domains.

Summary updated

Abstract

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.
BibTeX
@inproceedings{gururangan2022demix,
  author = {Suchin Gururangan and Mike Lewis and Ari Holtzman and Noah A. Smith and Luke Zettlemoyer},
  title = {{DEMix Layers: Disentangling Domains for Modular Language Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=1ljcCyJWN2h},
}
Figure