TL;DR
This paper investigates the effectiveness of converting constituency-based syntactic parses to dependency-based ones for Hungarian, a morphologically rich, free-word-order language. They found that while automatic conversion yields competitive accuracy, the types of errors made by parsers trained on converted data differ significantly from those trained on manually annotated "gold standard" data, highlighting the continued importance of manual annotation.
What Problem Does This Paper Address?
Syntactic parsing, the process of analyzing the grammatical structure of sentences, typically falls into two main camps: constituency parsing and dependency parsing. Constituency parsing breaks sentences down into hierarchical phrases, while dependency parsing focuses on the relationships between individual words. For many languages, dependency treebanks are created by automatically converting existing constituency treebanks. However, the quality and impact of this automatic conversion process, especially for languages with complex morphology and flexible word order like Hungarian, are not well understood. This paper aims to empirically evaluate this conversion process, comparing it to parsing with manually annotated data and exploring the implications for parser performance and error patterns.
Methods
The authors employed a rule-based system to convert constituency trees to dependency trees. They identified head words of clauses and established relationships between them (ROOT, COORD, ATT) based on grammatical roles and morphological information. For evaluation, they trained a dependency parser (Bohnet, 2010) on both manually annotated ("gold standard") dependency trees and automatically converted ("silver standard") dependency trees. They also experimented with training a constituency parser and then converting its output to a dependency format. Key metrics used for evaluation were Labeled Attachment Score (LAS) and Unlabeled Attachment Score (ULA), excluding punctuation.
Datasets
The primary datasets used were:
- Szeged Treebank: A manually annotated constituency treebank for Hungarian, containing 82,000 sentences.
- Szeged Dependency Treebank: A manually annotated dependency treebank for Hungarian, containing annotations for the same texts as the constituency treebank.
- Short business news domain: A subset of the Szeged Treebank used for detailed error analysis.
Key Insights and Ideas
- Manual annotation still matters: Despite competitive accuracy scores from automatically converted data, training on gold-standard dependency trees consistently yields better results. This suggests that automatic conversion, while useful, doesn't fully capture the nuances of human annotation.
- Error types differ: Parsers trained on automatically converted data make different kinds of mistakes compared to those trained on gold-standard data. This implies that the conversion process introduces specific biases or loses certain information that affects parsing decisions.
- Hungarian's complexity: The paper investigates whether the advantages of constituency parsers in configurational languages (like English) extend to Hungarian, a free-word-order language. Their findings suggest that the benefits of post-conversion (constituency to dependency) are not solely tied to configurational properties.
- Conversion challenges: Specific linguistic phenomena, such as coordination, multiple modifiers, and the attachment of conjunctions/adverbs, pose challenges for both conversion and parsing, regardless of the training data.
Key Results
- Conversion Accuracy: The automatic conversion of constituency trees to dependency trees achieved high accuracy: 96.51% ULA and 93.85% LAS.
- Parser Performance: Parsers trained on gold-standard dependency data outperformed those trained on silver-standard data by 1.6% ULA and 3.16% LAS.
- Post-conversion vs. Pre-conversion: Converting constituency parser output to dependency format (post-conversion) achieved competitive ULA scores (92.78%) compared to training on automatically converted dependency data (93.23%).
Limitations and Future Work
- Limitations: The authors acknowledge that even with automatic conversion, certain grammatical phenomena (e.g., conjunctions, adverbs) still require manual checking. They also note that the constituency treebank lacks annotations for some grammatical relations, which can lead to noisy training data for the silver standard.
- Future Work: The authors plan to further investigate how the advantages of constituency and dependency representations can be exploited in Hungarian parsing and intend to conduct further uptraining experiments with both types of parsers.
Summary updated