Using Foreign Inclusion Detection to Improve Parsing Performance

Beatrice Alex, Amit Dubey, Frank KellerEMNLP 2007

TL;DR

This paper tackles the challenge of English words and phrases appearing in German text, which significantly degrades the performance of standard NLP parsers. The authors introduce an annotation-free method for detecting these "foreign inclusions" and demonstrate that integrating this detection into a German parser, particularly by treating inclusions as distinct phrases, substantially improves parsing accuracy.

What Problem Does This Paper Address?

As English becomes a global language, its words and phrases are increasingly borrowed into other languages, a phenomenon known as language mixing or code-switching. This is particularly prevalent in technical, commercial, and media domains. While this is a common occurrence, most Natural Language Processing (NLP) tools, including parsers, are designed assuming monolingual input. When these tools encounter foreign words or phrases (like English "Anglicisms" in German text), they often fail to process them correctly. This can lead to unknown words, incorrect part-of-speech (POS) tagging, and crucially, incorrect phrase boundary detection, all of which severely degrade parsing accuracy. The paper aims to investigate the extent of this problem and propose a solution to mitigate it.

Methods

The core of the paper's contribution lies in two main areas:

  • Annotation-Free Foreign Inclusion Detection: The authors leverage a previously developed classifier that identifies English inclusions without requiring any manually annotated data. This classifier uses a combination of lexicon lookups, search engine queries, and a rule-based post-processing step to identify foreign words and phrases.
  • Parser Integration Strategies: To improve parsing performance, the detected inclusions are integrated into a state-of-the-art German parser in two ways:
  • Word-by-word Model: This approach simply marks detected inclusions with a special "FOM" (foreign material) tag. The parser is then restricted to assigning POS tags that have been observed in English inclusions during training.
  • Inclusion Entity Model: This more advanced method employs a tree transformation. It introduces a new "FP" (foreign phrase) category that groups consecutive tokens marked as FOM. This encourages the parser to treat multi-word inclusions as single constituent units, preventing incorrect phrase boundaries within them.

Datasets

  • TIGER Corpus: This is a large, bracketed German corpus of newspaper text (40,020 sentences) used for training and evaluating the parser.
  • Inclusion Set: A subset of the TIGER corpus containing sentences identified by the inclusion classifier as having multi-word English inclusions. This set was specifically curated for the experiments.
  • Random Set: A stratified sample of sentences from the TIGER corpus, matched in sentence length to the inclusion set, to serve as a control group.

Key Insights and Ideas

  • Foreign inclusions are a significant parsing bottleneck: Even a state-of-the-art German parser struggles considerably with sentences containing English inclusions, performing worse than on length-matched monolingual sentences.
  • Perfect POS tagging is insufficient: Simply providing the parser with correct POS tags for foreign words doesn't fully solve the problem; the structural interpretation of these inclusions is key.
  • Treating inclusions as phrases is crucial: The "inclusion entity model," which groups foreign words into phrases, significantly outperforms both a baseline parser and a parser with perfect POS tagging. This highlights the importance of recognizing the phrasal nature of many foreign borrowings.
  • Annotation-free detection is practical: The ability to detect inclusions without manual annotation makes the approach scalable and applicable to large, unannotated datasets.

Key Results

  • The inclusion detection classifier achieves an F-score approaching 85% across different domains.
  • The baseline parser achieves an F-score of 59.2 on the inclusion set, compared to 65.2 on the random set.
  • Providing perfect POS tags improves performance but still leaves a significant gap (F-score of 62.2 vs. 67.3).
  • The Inclusion Entity Model achieves an F-score of 63.5 on the inclusion set, a 4.3-point improvement over the baseline.
  • This model even outperforms a parser with perfect POS tagging on the inclusion set (63.5 vs. 62.2 F-score).
  • The inclusion entity model shows minimal degradation in performance on the random (monolingual) set, indicating it doesn't negatively impact parsing of standard sentences.

Limitations and Future Work

  • Inclusions remain harder to parse: Even with the best-performing model, sentences with English inclusions are still parsed less accurately than length-matched monolingual sentences. This suggests an inherent difficulty that might not be fully overcome by current methods.
  • Ambiguity in gold standard annotations: The paper notes that the gold standard annotation guidelines for differentiating phrase categories of inclusions could be improved, as there's ambiguity between categories like PN and CH.
  • Future work: The authors suggest exploring the application of their inclusion detection method to rule-based parsing and further investigating the impact of different types of inclusions and their interaction with parsing. They also mention the possibility of extending the inclusion classifier to new languages.

Summary updated

BibTeX
@inproceedings{alex2007using,
  author = {Beatrice Alex and Amit Dubey and Frank Keller},
  title = {{Using Foreign Inclusion Detection to Improve Parsing Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2007},
}
Figure