Predicting the Success of Domain Adaptation in Text Similarity

Nick Pogrebnyakov, Shohreh ShaghaghianACL 2021

TL;DR

This paper tackles the challenge of predicting how well a model trained on one text dataset will perform on another (domain adaptation). They propose using descriptive features of the datasets themselves, along with cross-domain similarity metrics, to build models that can either predict success or rank potential source domains for a given target domain. This work is crucial for making domain adaptation more efficient and scalable in practice.

What Problem Does This Paper Address?

Deep learning models have revolutionized Natural Language Processing (NLP), but they often require large amounts of labeled data. When applying a model trained on one dataset (source domain) to a different dataset (target domain) with a different data distribution, performance often degrades. This phenomenon is known as domain shift, and the process of adapting a model to a new domain is called domain adaptation (DA). While many DA techniques exist, a significant practical challenge is knowing in advance which source domain will yield the best results for a given target domain. Trying all possible source domains is computationally expensive and not scalable. This paper aims to bridge this gap by developing methods to predict or rank the success of domain transfer (using a model directly) and domain adaptation.

Methods

The authors propose two main approaches to predict domain adaptation success:

  • Success Predictor: This approach trains a binary classifier to predict whether a given source-target domain pair will achieve at least 80% of the performance of a fully supervised model trained on the target domain.
  • Domain Ranker: This approach models the problem as a ranking task, training a binary classifier to determine which of two source domains is better for a specific target domain.

Both approaches utilize a set of descriptive features derived from the source and target domains. These features include:

  • Unigram Coverage: The ratio of common unigrams between source and target domains.
  • Dataset Size: The number of labeled data points and average token count per example.
  • Distribution Similarity: Measures like Rényi divergence and KL divergence to capture how token distributions differ.
  • Language Similarity: Perplexity of a trigram language model trained on one domain evaluated on the other, and word vector variance.

For the domain adaptation experiments, they employ Stacked Denoising Autoencoders (SDA) and a more advanced version called Marginalized SDA with Domain Regularization (mSDAR). Text is first embedded using the Universal Sentence Encoder (USE), then corrupted with noise, and finally reconstructed to learn latent representations that align feature spaces. For domain transfer (DT), USE representations are used directly. The performance is measured by the F1-score, normalized by the in-domain performance to account for inherent dataset difficulty.

Datasets

The study uses 11 publicly available semantic text similarity datasets, treated as individual domains:

  • StackExchange Forums (7 datasets): Apple, AskUbuntu, Math, StackOverflow, Stats, SuperUser, and Unix.
  • Other datasets: Quora Question Pairs, Microsoft Research Paraphrase Corpus (MRPC), Paraphrase Adversaries from Word Scrambling (PAWS), and Sentences Involving Compositional Knowledge-Relatedness (SICK).

All datasets contain binary labels indicating similarity, with SICK's relevance score converted to a binary format.

Key Insights and Ideas

  • Domain characteristics are predictive: The paper demonstrates that intrinsic properties of text datasets (like vocabulary overlap, token distribution, and language model perplexity) can be strong indicators of domain adaptation success.
  • Domain Ranker outperforms Success Predictor: The Domain Ranker approach, which focuses on relative ordering of source domains, generally achieves better performance than the Success Predictor, which aims for an absolute success threshold. This suggests that relative comparisons are more robust.
  • PAWS and Quora are challenging: These datasets proved particularly difficult for the proposed methods, highlighting the need for more sophisticated domain similarity features that can capture complex dataset characteristics.
  • mSDAR generally outperforms SDA: Consistent with prior work, the mSDAR technique for domain adaptation shows slightly better results than standard SDA.

Key Results

  • Domain Ranker: Achieved an average Correct Rank Percentage (CRP) of 0.38 for DT, 0.45 for SDA, and 0.38 for mSDAR. The Top1 accuracy (correctly identifying the best source domain) was also respectable, averaging 0.64 for DT, 0.79 for SDA, and 0.79 for mSDAR.
  • Success Predictor: Showed more variability, with good performance on domains like Apple and AskUbuntu but poor performance on PAWS and Quora.
  • Cross-domain performance: Generally lower than in-domain performance, as expected. However, for PAWS, DA (both SDA and mSDAR) delivered significantly better performance than in-domain training.
  • Adaptation Successes: For mSDAR, an average of 4 adaptation successes were observed per target domain, indicating that in many cases, a source domain could be found that met the 80% performance threshold. PAWS was an exception, with all source domains succeeding. Quora was another exception, with no successes.

Limitations and Future Work

  • Limited dataset complexity: The authors acknowledge that their domain similarity features might not fully capture the complexities of datasets like PAWS and Quora, leading to poor prediction performance on these specific domains.
  • Fixed success threshold: The 80% threshold for success was fixed. Future work could explore learning this threshold dynamically.
  • Specific NLP task: The experiments focused on text similarity and autoencoder-based DA. Extending these methods to other NLP tasks and unsupervised DA approaches is a natural next step.
  • Multi-source adaptation: Investigating source selection in multi-source domain adaptation setups is a promising avenue.
  • Alternative representations: Exploring different text representations (e.g., bag-of-words, Transformer-based models) and models could further improve performance.

Summary updated

BibTeX
@inproceedings{pogrebnyakov2021predicting,
  author = {Nick Pogrebnyakov and Shohreh Shaghaghian},
  title = {{Predicting the Success of Domain Adaptation in Text Similarity}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure