Multidomain Pretrained Language Models for Green NLP

Antonis Maronikolakis, Hinrich SchützeOpenReview 2021 · OpenReview

TL;DR

This paper introduces the concept of "multidomain" language models, which can be adapted to multiple domains simultaneously with minimal performance loss compared to single-domain models. This approach significantly reduces computational costs and data requirements, making advanced NLP more accessible and environmentally friendly.

What Problem Does This Paper Address?

Traditionally, to achieve optimal performance on a specific NLP task within a particular domain (e.g., medical text analysis), researchers adapt a pre-trained language model using unlabeled data from that domain before fine-tuning it on the labeled task data. While effective, this process has a major drawback: if you want to tackle tasks across N different domains, you need N separate adapted models. This is computationally expensive, requires substantial storage, and becomes prohibitive for low-resource environments or devices. The paper aims to solve this by demonstrating that a single model can be adapted to multiple domains concurrently, offering a more efficient and sustainable solution.

Methods

The core idea is to extend the concept of domain adaptation to handle multiple domains at once. The authors explore this by adapting a DistilBERT model, a smaller and faster version of BERT, to several domains simultaneously. They compare this "multidomain" model against:

  • Base Model: A standard pre-trained DistilBERT model without any domain adaptation.
  • Single-Domain Models: DistilBERT models adapted individually to each of the four domains used in the study.

The adaptation process involves training on raw, unlabeled text data from each domain. The key innovation is adapting a single model across all domains, either sequentially or jointly. They also investigate the impact of the order of domain adaptation and whether adapting to irrelevant domains is beneficial (finding it is not).

Datasets

The paper uses the following datasets for domain adaptation and evaluation:

  • Domain Adaptation Data:
    • Amazon (reviews across various product categories)
    • Arxiv (scientific papers)
    • Realnews (news articles)
    • CS (computer science papers)
    • Reddit Comments
    • PubMed (for low-resource domain experiments)
  • Supervised Task Evaluation Data:
    • ACL-ARC (citation intent classification)
    • SciCite (citation intent classification)
    • SARC (sarcasm detection)
    • AG-News (news topic classification)
    • IMDB (sentiment analysis)
    • Clothing Reviews (sentiment analysis)
    • ChemProt (protein-chemical relation extraction)
    • PubMed-RCT (biomedical relation extraction)
    • MultiNLI (natural language inference)

Key Insights and Ideas

  • Multidomain Adaptation is Viable: A single model can be adapted to multiple domains without significant performance degradation compared to specialized single-domain models.
  • Efficiency Gains: This approach drastically reduces the need for multiple models, saving computational resources, storage, and training time. This is crucial for "green NLP" and for deploying models on resource-constrained devices.
  • Low-Resource Benefits: Similar to multilingual models, multidomain models can boost performance on low-resource domains by leveraging the knowledge gained from larger, more data-rich domains.
  • Domain Adaptation Order Doesn't Matter Much: The order in which domains are adapted to a single model has a surprisingly small impact on overall performance.
  • Irrelevant Domain Adaptation is Harmful: Pretraining on an irrelevant domain does not improve performance and can even slightly degrade it.

Key Results

  • Performance: The multidomain model achieved performance comparable to single-domain models across eight tasks and the MultiNLI dataset. On average, the multidomain model showed a 1.4% improvement over the base model, while single-domain models showed a 2.0% improvement. Crucially, the multidomain model never performed worse than the base model.
  • Data Efficiency: Training a multidomain model required only half the data (8GB total) compared to training four separate single-domain models (16GB total).
  • Low-Resource Domain Improvement: Adapting the multidomain model to a low-resource biomedical domain (PubMed) further improved performance on related tasks, outperforming models trained solely on that domain.
  • Domain Adaptation Order: The average performance across all tested domain adaptation orders was 81.4%, with a minimum of 81.1% and a maximum of 81.7%, showing robustness to order.

Limitations and Future Work

  • Order of Adaptation: While the paper shows that the order of adaptation has minimal impact, it's noted that the last adapted domain might have a slightly larger effect, though this correlation wasn't strongly observed.
  • Joint vs. Sequential Adaptation: The paper found no substantial difference between jointly training on all domains or adapting sequentially. This suggests flexibility in implementation.
  • Future Work: The authors suggest that these multidomain models are well-suited for low-resource devices due to their efficiency. They also highlight the potential for further exploration in adapting to more domains and investigating the impact of dataset size within each domain. The work also paves the way for more environmentally conscious NLP research and development.

Summary updated

Abstract

When tackling a task in a given domain, it has been shown that adapting a model to the domain using raw text data before training on the supervised task improves performance versus solely training on the task. The downside is that a lot of domain data is required and if we want to tackle tasks in n domains, we require n models each adapted on domain data before task learning. Storing and using these models separately can be prohibitive for low-end devices. In this paper we show that domain adaptation can be generalised to cover multiple domains. Specifically, a single model can be trained across various domains at the same time with minimal drop in performance, even when we use less data and resources. Thus, instead of training multiple models, we can train a single multidomain model saving on computational resources and training time.
BibTeX
@misc{maronikolakis2021multidomain,
  author = {Antonis Maronikolakis and Hinrich Schütze},
  title = {{Multidomain Pretrained Language Models for Green NLP}},
  year = {2021},
  url = {https://openreview.net/forum?id=gYw0s9y6Ozbq},
  howpublished = {OpenReview},
}
Figure