Automatic Domain Adaptation for Parsing

David McClosky, Eugene Charniak, Mark JohnsonNAACL 2010

TL;DR

Statistical parsers struggle to generalize across different text domains. This paper introduces "multiple source parser adaptation," a new task where parsers learn to combine models from various domains to achieve better performance on unseen target texts. Their novel approach uses a regression model to predict the best combination of source parsers, outperforming all non-oracle baselines.

What Problem Does This Paper Address?

Statistical parsers, trained on specific datasets (domains), often see a significant drop in performance when applied to text from a different domain. This is a major hurdle for applications that need to process diverse text, like parsing the entire web. While some work has explored adapting parsers, determining the optimal combination of pre-trained models for a new, unseen domain remains a challenge. This paper tackles this by framing it as "multiple source parser adaptation," where the goal is to automatically select and combine parsing models from a pool of diverse source domains to achieve the best possible performance on a target text, without prior knowledge of the target domain's identity.

Methods

The core of the paper's contribution lies in its novel approach to domain adaptation. Instead of training a single, monolithic parser, they propose a system that leverages multiple pre-trained parsers from various source domains. The key innovation is a domain-aware linear regression model. This model takes features that describe the differences between a target text and each source domain (e.g., word frequency overlap, unknown words) and predicts the expected parsing accuracy (f-score) for a combined model.

To create these combined models, they use linear interpolation of parsing model statistics from the selected source domains. The regression model is trained on sampled mixtures of source domains and their actual f-scores on target texts. This allows the system to learn which combinations of source parsers are likely to perform best on a given target text. The features used are:

  • COSINETOP50: Measures similarity based on the overlap of the top 50 most frequent words.
  • UNKWORDS: Measures the percentage of words in a source domain not seen in the target domain, focusing on low-frequency words.
  • ENTROPY: Measures the uniformity of the distribution over source domains, acting as a regularizer.

Datasets

The paper utilizes a variety of corpora for training and evaluation, aiming for diverse domains:

  • Source Domains: WSJ (news), NANC (news), ETT (broadcast news translation), BROWN (literature), GENIA (biomedical abstracts), MEDLINE (biomedical abstracts, two versions), SWBD (spontaneous speech), BNC (British National Corpus).
  • Self-trained Corpora: Automatically parsed corpora derived from the above, used to augment training data and improve robustness.
  • Target Domains: Six domains were used for evaluation, with specific schemas for out-of-domain and in-domain evaluation scenarios.

The paper introduces the task of "multiple source parser adaptation" and provides an evaluation framework for it.

Key Insights and Ideas

  • Domain-specific parsers are good, but combining them is better: The paper strongly suggests that a single parser, even one trained on a large, general corpus like WSJ, is suboptimal for diverse text. However, simply mixing all available corpora uniformly isn't ideal either. The sweet spot lies in intelligently combining domain-specific knowledge.
  • Measuring domain differences is crucial: The success of their approach hinges on quantifying how different a target text is from potential source domains. Features like word frequency overlap and unknown words prove to be effective indicators.
  • Regression as a predictor of parsing performance: Using a regression model to predict f-scores for parser mixtures is a powerful way to guide the selection of the best combination without needing to exhaustively test every possibility.
  • Self-training is a significant enabler: The inclusion of self-trained corpora (automatically parsed data) dramatically improves performance, highlighting its importance in reducing data sparsity and adapting to new domains.

Key Results

The paper demonstrates that their proposed "multiple source parser adaptation" system significantly outperforms various baselines.

  • Out-of-domain evaluation: Their model achieved an average f-score of 84.0 ± 2.5, which was only 0.3% worse than the "BEST SEEN" oracle baseline (84.3 ± 2.6). This indicates a strong ability to adapt to entirely new domains.
  • In-domain evaluation: Their model achieved an average f-score of 86.9 ± 2.4, only 0.6% worse than the "BEST SEEN" oracle baseline (87.5 ± 2.1).
  • Outperforming fixed models: Their system significantly beat the "FIXED SET: WSJ" baseline (82.0 ± 4.8), showing the limitations of using a single, general-purpose parser.
  • Value of customization: Their model was 0.7% better than the "BEST OVERALL MODEL" (86.2 ± 1.9), which represents the single best model across all target domains. This proves the value of customizing parsing models to specific target domains.

Limitations and Future Work

  • Feature Engineering: While their chosen features (COSINETOP50, UNKWORDS, ENTROPY) are effective, the paper acknowledges that exploring a wider range of features could further improve performance.
  • Parser Independence: The current work focuses on the Charniak (2000) generative parser. Applying these methods to other parsers, especially discriminative rerankers, is a direction for future work.
  • Joint Domain and Syntax Estimation: The authors suggest that separating domain and syntax estimation is a useful step, but future work could explore learning these jointly, potentially combining their approach with methods that explicitly model domain-specific and general syntactic properties.
  • Scalability: While the paper demonstrates effectiveness, scaling this approach to the vastness of the web, where every page might be a unique domain, remains a significant challenge.

Summary updated

BibTeX
@inproceedings{mcclosky2010automatic,
  author = {David McClosky and Eugene Charniak and Mark Johnson},
  title = {{Automatic Domain Adaptation for Parsing}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}

Loading related content...

Figure