Adapting the RASP System for the CoNLL07 Domain-Adaptation Task

Rebecca Watson, Ted BriscoeEMNLP 2007

TL;DR

This paper tackles the challenge of adapting a dependency parser (RASP) trained on one linguistic domain (general English) to a new domain (biomedical text and child language) with different annotation schemes. They found that mapping between annotation schemes was a major hurdle, but a semi-supervised bootstrapping approach, especially when leveraging unlabeled data, showed significant performance improvements on the target domain.

What Problem Does This Paper Address?

Natural Language Processing (NLP) models often perform best when trained on data that closely matches the domain they will be used in. However, acquiring large amounts of labeled data for every specific domain (like medical literature or specialized scientific texts) is expensive and time-consuming. This paper addresses the problem of domain adaptation: how can we take a parser that works well on general text and make it perform well on a different type of text without retraining it from scratch on a massive new dataset? Specifically, they focus on adapting the RASP parser, which uses a grammar-based approach to dependency parsing, to the CoNLL07 shared task's domain adaptation challenge, which involved adapting parsers to biomedical and child language corpora. A key challenge here is that these new domains use a different annotation scheme (dependency relations, DRs) than RASP's native scheme (grammatical relations, GRs), requiring a complex mapping.

Methods

The core of their approach involves bootstrapping and semi-supervised learning. They start with the RASP system, which is a generalized LR parser that builds a packed parse forest.

  1. Mapping Annotation Schemes: The most crucial initial step was mapping RASP's grammatical relations (GRs) to the task's dependency relations (DRs). This was a non-trivial process due to fundamental differences in how linguistic phenomena were represented.
  2. Initial Model Training: They used the DR-annotated PTB (Penn Treebank) training data. They mapped this data to their GR scheme and then identified sentences where RASP could produce a single, unambiguous derivation consistent with the mapped GRs. This subset of unambiguous data was used to train an initial parse ranking model (Model B).
  3. Confidence-Based Self-Training: For sentences with multiple possible derivations (ambiguous data), they used the initial model to rank these derivations. They then trained a new model (Model C) using weighted counts from these consistent derivations, where weights reflected their confidence in the derivation's probability.
  4. Domain Adaptation (Bootstrapping): To adapt to the new domains (biomedical and CHILDES), they used the already trained model (Model C) to find the highest-ranked parse for the small amount of in-domain annotated data. This was combined with the existing training data to create a new model (Model D).
  5. Unsupervised Adaptation: They then iteratively applied this process to the unannotated in-domain data. Model D was used to rank parses for the unlabeled data, and these top-ranked parses were used to retrain the model (leading to Models E, F, and G). This allowed them to leverage large amounts of unlabeled data for adaptation.
  6. POS Tagging: Since RASP requires input in a specific Part-of-Speech (POS) tagset (CLAWS), they used a first-order HMM POS tagger trained on general corpora for their submission.

Datasets

  • Training Data:
    • PTB (Penn Treebank): Re-annotated with dependency relations (DRs). This served as the primary source for initial model training.
  • PBIOTB (Biomedical Sentences): A small set of 200 annotated biomedical sentences used for in-domain adaptation.
  • CHILDES Database: Annotated with DRs, used for adapting to child language data.
  • Test Data:
    • Biomedical Articles Corpus: Provided as test data for the biomedical domain.
  • CHILDES Database: Provided as test data for the child language domain.
  • Evaluation Data:
    • DepBank: A reannotated subset of the WSJ PTB used for evaluating intermediate models with their own GR-based evaluation scheme.
  • Blind Test Data: The official test sets for the biomedical and CHILDES domains.

Key Insights and Ideas

  • Annotation Scheme Mismatch is a Major Hurdle: The most significant challenge was reconciling the DR annotation scheme with RASP's GR scheme. Differences in linguistic assumptions, the number of distinctions, and the representation of phenomena like punctuation and auxiliary verbs created substantial noise and difficulty in mapping.
  • Bootstrapping with Unlabeled Data is Powerful: The semi-supervised bootstrapping approach, especially when extended to use unlabeled in-domain data, proved effective. It allowed the system to learn from the target domain's characteristics even without explicit annotations for all sentences.
  • Confidence-Based Weighting Matters: Using confidence scores to weight derivations during retraining helped to prioritize more reliable parses and improve model performance.
  • Unlabeled Dependencies Can Be More Robust: For the official submission, they found that using unlabeled dependencies (ignoring the specific relation types) for bootstrapping worked better than using the derived labeled mapping, highlighting the difficulty and potential inaccuracies of the mapping process.
  • Punctuation Representation is Tricky: The differing treatment of punctuation between the GR and DR schemes was a specific example of the annotation mismatch problem, significantly impacting evaluation metrics.

Key Results

  • Significant Improvement on PCHEMTB: For the biomedical domain (PCHEMTB), the adapted model (Model G) showed a statistically significant improvement over the unadapted model. The paper reports an improvement of over 7% in F1 score when factoring out the effect of P relations.
  • Limited Adaptation for CHILDES: For the CHILDES data, the results showed no significant evidence of adaptation to the domain.
  • Intermediate Model Performance: Table 1 shows a steady increase in performance (Precision, Recall, F1) as successive bootstrapping models (A through G) were trained on the PTB data. Model G achieved an F1 score of 76.41 on the PTB test data.
  • Official Scores: Model G achieved an F1 score of 72.62 on the PCHEMTB test data, and Model H achieved 72.88 on the CHILDES test data (Table 3).

Limitations and Future Work

  • Mapping Noise: The mapping procedure between the DR and GR schemes introduced considerable noise into the training process, as evidenced by the average F1 score of only 84.56 between mapped GRs and RASP output for the PTB data.
  • Time and Information Constraints: The lack of detailed information about the DR scheme and limited time hampered their ability to derive an optimal mapping.
  • Unlexicalized Model: The decision to use an entirely unlexicalized parse ranking model might have limited performance compared to a lexicalized approach.
  • Future Work: The authors suggest that further work on improving the mapping between annotation schemes and exploring more sophisticated adaptation techniques could yield even better results. The difficulty with punctuation representation also points to areas for future research.

Summary updated

BibTeX
@inproceedings{watson2007adapting,
  author = {Rebecca Watson and Ted Briscoe},
  title = {{Adapting the RASP System for the CoNLL07 Domain-Adaptation Task}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure