DEMix Layers: Disentangling Domains for Modular Language Modeling

Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, Luke Zettlemoyer

Abstract

Standard dense language models (LMs) update all parameters for every token, leading to computational inefficiency, domain dilution, and inflexibility. We introduce DEMix (Domain Expert Mixture) layers, which replace standard feedforward layers with a collection of domain-specialized experts. By routing data based on its source during training and employing a parameter-free Bayesian mixing strategy at inference, DEMix models achieve superior generalization and efficiency. Furthermore, this modularity allows for the addition or removal of domain-specific knowledge post-training without retraining the entire model or suffering from catastrophic forgetting.

Introduction

Modern large-scale language models are typically "dense," meaning every parameter is active for every input. This approach suffers from several limitations. First, it is computationally expensive to update billions of parameters for every token. Second, dense models often favor dominant domains in the training data, resulting in poor performance on niche or low-resource topics. Finally, these models are inflexible; adapting to new domains often causes "catastrophic forgetting" of previous knowledge, and removing unwanted knowledge (e.g., toxic content) is nearly impossible without full retraining.

The DEMix Architecture

The core of our approach is the DEMix layer, a drop-in replacement for the feedforward network (FFN) layers in standard Transformer architectures.

The DEMix layer architecture supports modular training, inference-time mixing, and the addition or removal of domain experts.

Modular Training and Routing

Instead of a single FFN, a DEMix layer contains $n$ "experts," each corresponding to a specific training domain. During training, the model uses document metadata to route tokens to the appropriate expert. For instance, tokens from medical journals are processed only by the "Medical Expert." This ensures that domain-specific knowledge is isolated within dedicated parameters.

Inference-Time Mixing

When the domain of input text is unknown at test time, we use a parameter-free probabilistic approach. The model estimates the posterior probability $P(D_t | x_{<t})$ of which domain the text belongs to and computes a weighted ensemble of the experts' outputs. This allows the model to handle heterogeneous text—such as a legal analysis of medical data—by combining relevant experts.

Domain-Adaptive Pretraining (DEMix-DAPT)

To adapt to a novel domain, we simply add a new expert to the DEMix layer. This new expert is initialized from the most similar existing expert and trained exclusively on the new data while all other parameters remain frozen. This process prevents catastrophic forgetting of the original training domains.

Experimental Setup

We evaluated DEMix using a Multi-Domain Corpus consisting of 73.8 billion tokens across eight training domains: News (1B), Computer Science, Legal, Medical, WebText, RealNews, Reddit, and Amazon Reviews. We also tested generalization on eight novel domains, including ACL papers, Github code, and CORD-19 research.

Results and Discussion

DEMix layers consistently reduced test-time perplexity across all model sizes. Notably, a 1.3B parameter DEMix model achieved zero-shot results on the Penn TreeBank benchmark comparable to the 175B parameter GPT-3.

Domain Identification

The Bayesian mixing strategy effectively identifies the correct domain during inference. As shown in the heatmap below, the model assigns high probability to the correct training domain expert when processing relevant text.

Heatmap showing the posterior probability $P(D_t | x_{<t})$ across training domains, demonstrating the model's ability to correctly identify the source domain at inference time.

Efficiency and Modularity

While DEMix models have more total parameters, the number of active parameters per token remains constant, maintaining FLOP efficiency. Furthermore, the disentangled nature of the experts allows for "plug-and-play" modularity. Removing an expert significantly degrades performance on its specific domain while leaving other domains unaffected, providing a mechanism for safer AI deployment and targeted knowledge management.

Conclusion

DEMix layers offer a path toward more efficient, interpretable, and flexible language models. By leveraging natural domain boundaries, we achieve state-of-the-art performance while enabling modular updates and domain-specific control. Future work will explore unsupervised domain discovery to remove the reliance on metadata.

Summary updated

BibTeX
@inproceedings{gururangan2022demix,
  author = {Suchin Gururangan and Mike Lewis and Ari Holtzman and Noah A. Smith and Luke Zettlemoyer},
  title = {{DEMix Layers: Disentangling Domains for Modular Language Modeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=1ljcCyJWN2h},
}
Figure