Exploring an Auxiliary Distribution Based Approach to Domain Adaptation of a Syntactic Disambiguation Model

Barbara Plank, Gertjan van NoordCF+CDPE @ COLING 2008 · OpenReview

Abstract

Barbara Plank, Gertjan van Noord. Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation.

2008.

Introduction

Statistical parsers typically exhibit a significant drop in performance when applied to domains differing from their training data. This "domain adaptation" problem is particularly acute for parsers trained on formal newspaper text, such as the Dutch Alpino corpus, when tasked with processing informal spoken language or specialized question-answering (Q&A) datasets. While manual annotation of target-domain data is the ideal solution, it is often prohibitively expensive. This paper investigates the effectiveness of "auxiliary distributions"—a technique originally used to incorporate lexical preferences—as a method for supervised domain adaptation for the Alpino parser.

Methodology

The study operates within the Maximum Entropy (MaxEnt) framework, where a model selects the most probable parse by weighting various linguistic features. The authors compare two primary adaptation strategies:

  1. Auxiliary Distribution Approach: A pre-trained, large-scale "out-of-domain" model is integrated into a new model trained on a small target dataset. The out-of-domain model's output (specifically, the negative log-probability of a parse) is treated as a single, real-valued feature within the new MaxEnt model. This allows the learner to theoretically re-weight general domain preferences alongside new in-domain evidence.
  2. Model Combination: A simpler baseline where the final scores of an in-domain model and an out-of-domain model are linearly interpolated or multiplied (e.g., using equal weights).

Datasets

Three primary Dutch text sources were utilized:

  • Alpino Treebank (Source Domain): Approximately 7,000 sentences of newspaper text.
  • CLEF Treebank (Target Domain 1): Approximately 1,800 questions from a Question Answering task.
  • Spoken Dutch Corpus (CGN) (Target Domain 2): A diverse set of spoken data, ranging from spontaneous conversations to formal speeches.

Results and Discussion

Performance on Q&A Data

On the CLEF 2004 dataset, the "in-domain only" model achieved a Concept Accuracy (CA) of 96.60%. Surprisingly, the addition of an auxiliary distribution failed to improve this baseline. In contrast, simple model combination with equal weights significantly boosted performance to 97.23%.

Performance comparison on CLEF 2004 data across varying amounts of training data.

As shown above, the model combination approach consistently outperforms the auxiliary distribution method across different training set sizes. The auxiliary distribution often struggles to surpass the performance of the out-of-domain model alone when training data is extremely limited.

The Impact of Data Scarcity

The authors identify a "weight estimation trap" when training data is scarce. In scenarios with only 1% to 10% of the target data, the MaxEnt learner often fails to accurately estimate the weight for the auxiliary feature.

Concept Accuracy results for small percentages of training data (0% to 10%).

The data in the table above illustrates this trend. At 1% training data, the auxiliary distribution approach ("+aux") frequently performs worse than the model combination ("m.c.") or even the out-of-domain baseline. The simpler interpolation of models provides a more robust "safety net" than the more complex feature-level integration.

Performance on Spoken Data

The results for the Spoken Dutch Corpus were even more pronounced. In subdomains like spontaneous face-to-face conversations, the auxiliary feature occasionally degraded performance compared to the baseline, reinforcing the finding that complexity does not guarantee gains in domain adaptation.

Conclusion

This investigation reveals that for the Alpino parser, the sophisticated auxiliary distribution approach often fails to beat simpler model combination techniques for domain adaptation. The primary bottleneck appears to be the difficulty of reliably estimating feature weights when in-domain training data is limited. For practical domain adaptation with small datasets, keeping models separate and combining their outputs remains a more effective and stable strategy.

Summary updated

Abstract

Barbara Plank, Gertjan van Noord. Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation. 2008.
BibTeX
@inproceedings{plank2008exploring,
  author = {Barbara Plank and Gertjan van Noord},
  title = {{Exploring an Auxiliary Distribution Based Approach to Domain Adaptation of a Syntactic Disambiguation Model}},
  booktitle = {CF+CDPE @ COLING},
  year = {2008},
  url = {https://openreview.net/forum?id=v2cQCijreB},
}
Figure