An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines

Anders SøgaardDepLing 2013 · OpenReview

Abstract: Anders Søgaard. Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013).

2013.

Introduction

In Natural Language Processing (NLP), dependency parsing identifies grammatical relationships between words—such as which noun is the subject of which verb. However, linguistic theory often diverges on "difficult cases." For instance, should a preposition be the "head" of a noun, or vice versa? Because many modern treebanks are automatically converted from older formats, several conversion schemes (Yamada-Matsumoto, Stanford, CoNLL, and LTH) have emerged. This study investigates whether these differences are merely academic or if they impact the performance of downstream AI applications.

Methods and Structural Divergence

The author compares four major conversion schemes using quantitative metrics like tree-edit distance (TED) and Unlabeled Attachment Score (UAS). Beyond standard distance measures, the study examines how each scheme handles specific linguistic ambiguities that define their structural identity.

Table 1: Head-dependent choices for difficult linguistic cases across different conversion schemes.

As shown in the table above, schemes differ significantly in their treatment of functional versus substantive heads. For example, while Yamada and CoNLL treat the auxiliary as the head of the main verb (indicated by '1'), Stanford and LTH reverse this relationship (indicated by '2').

To test the real-world impact of these choices, the author evaluates the schemes across five distinct NLP applications: Negation Resolution, Semantic Role Labeling (SRL), Statistical Machine Translation (SMT), Sentence Compression, and Perspective Classification. These tests utilize standard benchmarks including the Penn-III Treebank, Ontonotes 4.0, and the Europarl corpus.

Learnability vs. Utility

A central theme of the research is the distinction between how easily a parser can learn a scheme ("learnability") and how useful that scheme is for a specific task ("utility"). The study uses derivational perplexity (d-ppl) as a parser-independent metric to measure the inherent complexity of these structures.

Table 2: Perplexity scores across schemes, indicating inherent complexity and learnability.

The Yamada-Matsumoto scheme consistently shows the lowest d-ppl (520.8 in gold data), making it the most "learnable" for parsers. However, high learnability does not equate to universal superiority. While Yamada is robust across domains, it is often outperformed in specialized tasks by more complex schemes.

Key Insights and Results

  • Parser Bias as a Smoother: Parsers often act as "smoothers." Even when gold standard annotations differ wildly, the output of a trained parser tends to look more similar across schemes, effectively canceling out some complex, non-local differences.
  • Substantive Heads for Semantics: Schemes like LTH and Stanford, which prioritize meaning-heavy "substantive" heads, perform significantly better in Semantic Role Labeling (SRL) and Machine Translation. In translation, these schemes helped systems better learn "V2" word order in German, leading to improved BLEU scores.
  • Negation Success: The Yamada scheme performed best for Negation Resolution, suggesting its specific handling of auxiliary verbs and punctuation is highly compatible with identifying the scope of a negation.

Conclusion

The study concludes that there is no single "best" dependency representation. While some schemes like Yamada are more robust and easier to learn, others like LTH are better suited for semantically demanding tasks. The author suggests that the future of dependency parsing lies not in finding a one-size-fits-all standard, but in developing methods to optimize or combine annotations specifically for the end-task at hand.

Summary updated

Abstract

Anders Søgaard. Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013). 2013.
BibTeX
@inproceedings{sgaard2013empirical,
  author = {Anders Søgaard},
  title = {{An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines}},
  booktitle = {DepLing},
  year = {2013},
  url = {https://openreview.net/forum?id=OS9TeO4GdZD},
}
Figure