Multi-Task Supervised Pretraining for Neural Domain Adaptation

Sara Meftah, Nasredine Semmar, Mohamed-Ayoub Tahiri, Youssef Tamaazousti, Hassane Essafi, Fatiha SadatACL 2020

TL;DR

This paper introduces MuTSPad, a novel approach for neural domain adaptation in NLP that unifies multi-task learning and supervised pre-training. By training a hierarchical model across multiple tasks on a high-resource source domain and then fine-tuning it on a low-resource target domain, MuTSPad significantly improves performance on tasks like POS tagging, chunking, NER, and dependency parsing, especially for domains like social media.

What Problem Does This Paper Address?

Building effective NLP systems often requires large amounts of labeled data. However, obtaining such data is expensive and time-consuming, leading to "low-resource" domains where annotated data is scarce. This is particularly true for domains like social media, which are rich in information but lack sufficient labeled datasets for traditional NLP tasks. Two common approaches to tackle this are:

  1. Mono-Task Pre-training (MTP): Train a model on a large, labeled source domain and then fine-tune it on the limited data of the target domain.
  2. Multi-Task Learning (MTL): Train a single model to perform multiple related tasks simultaneously, leveraging shared knowledge to improve generalization.

While both methods have shown success, they have limitations. MTP can be task-specific, and MTL still struggles with data scarcity. This paper aims to bridge this gap by combining the strengths of both approaches to achieve better domain adaptation in low-resource settings.

Methods

The core of this paper is the Multi-Task Supervised Pre-training and Adaptation (MuTSPad) method. It's a hierarchical approach that combines supervised pre-training and multi-task learning:

  • Hierarchical Multi-Task Architecture: The model is designed with a shared component (Word Representation Extractor) and then branches out for each task (POS, CK, NER, DP). Crucially, the tasks are organized hierarchically, with lower-level tasks (like POS) feeding into higher-level ones (like CK, NER, DP), reflecting linguistic dependencies observed in prior work.
  • Multi-Task Supervised Pre-training: The hierarchical model is first trained on a comprehensive, multi-task dataset from a high-resource source domain (e.g., News). This allows the model to learn general linguistic features and task-specific knowledge from a wealth of data.
  • Fine-tuning on Target Domain: After pre-training, the model is fine-tuned on the low-resource target domain (e.g., Tweets), again using a multi-task setup. This adapts the learned representations to the specific characteristics of the target domain.
  • Datasets Unification: To address the challenge of heterogeneous datasets (where each dataset might only be labeled for one task), the authors propose a "Dataset Unification" strategy. This involves creating unified datasets (e.g., "EnglishAll" and "TweetAll") where each sentence is annotated for all tasks, either through original labels or automatically generated ones. This allows for classical multi-task learning during both pre-training and fine-tuning.

Datasets

The authors use two main domains:

  • Source Domain (EnglishAll):
    • POS: Penn Tree-Bank (PTB)
    • CK: CONLL-2000
    • NER: CONLL-2003
    • DP: UD-English-EWT
  • Target Domain (TweetAll):
    • POS: TweeBank
    • CK: TChunk
    • NER: WNUT-17
    • DP: TweeBank (with Universal Dependency relations)

They also introduce the concept of "Dataset Unification" to create comprehensive "EnglishAll" and "TweetAll" datasets, where each sentence is annotated for all four tasks, overcoming the heterogeneity of individual datasets.

Key Insights and Ideas

  • Synergy of Pre-training and Multi-Task Learning: The paper demonstrates that combining supervised pre-training on a rich source domain with multi-task learning on both source and target domains is more effective than using either approach in isolation for domain adaptation.
  • Hierarchical Task Structure Matters: Organizing tasks hierarchically (e.g., POS -> CK -> NER/DP) based on linguistic dependencies significantly benefits the learning process, allowing lower-level tasks to act as "universal helpers" for higher-level ones.
  • Dataset Unification Simplifies Heterogeneity: The proposed "Dataset Unification" method effectively tackles the challenge of working with multiple, individually labeled datasets by creating a single, multi-task annotated dataset, simplifying the training process.
  • Low-Level Tasks are Crucial for High-Level Tasks: Analysis shows that units within the POS task's representation are highly important for the performance of higher-level tasks like CK, NER, and DP, underscoring the value of the hierarchical structure.

Key Results

MuTSPad achieved state-of-the-art or competitive results across the evaluated tasks:

  • Overall Improvement: MuTSPad outperformed both Mono-Task Learning and Mono-Task Fine-Tuning baselines by approximately +26% and +10% respectively, based on the mNRG (median Normalized Relative Gain) metric.
  • Task-Specific Performance:
    • DP: Achieved the best SOTA score, outperforming others by ~2.5%.
  • POS, CK, NER: Achieved competitive scores, though slightly lower than some specialized SOTA methods that might leverage different pre-training strategies.
  • Impact of Unification: Pre-training on unified source datasets slightly improved performance. Fine-tuning on unified target datasets was beneficial for POS and DP but hurt NER performance, which the authors attribute to noisy automatic NER predictions in the target domain.

Limitations and Future Work

  • NER Performance with Unified Target Data: The paper notes that fine-tuning on unified target datasets hurt NER performance, suggesting issues with automatically generated NER labels in the TweetAll dataset.
  • Complementarity with Unsupervised Pre-training: The authors suggest that incorporating unsupervised pre-trained embeddings like ELMo into MuTSPad could lead to even higher performance, particularly for NER.
  • Future Directions:
    • Explore soft multi-task architectures.
  • Investigate combinations of supervised and unsupervised multi-tasking.
  • Apply MuTSPad's learned representations to higher-level NLP applications like machine translation and sentiment analysis.

Summary updated

BibTeX
@inproceedings{meftah2020multi,
  author = {Sara Meftah and Nasredine Semmar and Mohamed-Ayoub Tahiri and Youssef Tamaazousti and Hassane Essafi and Fatiha Sadat},
  title = {{Multi-Task Supervised Pretraining for Neural Domain Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure