Evaluating Universal Dependency Parser Recovery of Predicate Argument Structure via CompChain Analysis

Sagar Indurkhya, Beracah Yankama, Robert C. BerwickACL 2021

TL;DR

This paper introduces "compchains" as a novel metric to evaluate how well dependency parsers capture predicate-argument structure, especially when clauses are embedded within each other. They found that current parsers struggle with complex embeddings, and that existing evaluation metrics don't fully capture this limitation, suggesting a need for more nuanced evaluation and potentially larger, more diverse training datasets.

What Problem Does This Paper Address?

Accurately understanding the relationship between verbs (predicates) and their arguments (who did what to whom) is crucial for many Natural Language Processing (NLP) tasks, like information extraction and sentiment analysis. Universal Dependencies (UD) is a widely adopted standard for annotating dependency structures in text across many languages. However, evaluating how well a UD parser recovers this predicate-argument structure, particularly when sentences become complex with embedded clauses (e.g., "He knows that she can kick the ball"), is challenging. Existing evaluation metrics like UAS and LAS primarily focus on the correctness of syntactic attachments and don't specifically measure the recovery of these deeper semantic relationships, especially in the presence of embedding. This gap means we don't fully understand the limitations of current parsers in handling complex sentence structures.

Methods

The core innovation of this paper is the introduction of compchains. A compchain is defined as a sequence of specific dependency relations (primarily ccomp and xcomp) that trace the predicate-argument structure from the root of a dependency parse tree. Essentially, it's a way to categorize the "depth" and type of clausal embedding present in a parse. The authors then use compchain classification as a proxy for evaluating predicate-argument structure recovery. This involves checking if the compchain derived from a parser's output matches the compchain from the gold-standard annotation for a given sentence. If they match, the parser is considered to have correctly identified the predicate-argument structure with embedding for that sentence. They applied this method to evaluate baseline UDPipe dependency parsers on various UD treebanks.

Datasets

The study utilized several Universal Dependencies (UD) treebanks for evaluation:

  • English Treebanks:
    • English Web Treebank (EWT)
    • English-Swedish Parallel Treebank (LinES)
    • GUM Treebank
  • Multilingual Treebanks:
    • Distributions of compchains were analyzed across 61 UD treebanks for various languages.

The parsers evaluated were CoNLL'18 Shared Task baseline models (UDPipe v1.2) trained on these treebanks.

Key Insights and Ideas

  • Compchains as a Revealing Metric: The compchain metric effectively highlights a specific weakness in current UD parsers: their struggle with predicate-argument structures involving embedded clauses.
  • Sparsity of Complex Embeddings: UD treebanks, particularly in their test sets, are surprisingly sparse in sentences with deeply embedded predicate-argument structures (compchains of length 3 or more). This suggests that parsers might not be adequately trained or tested on these complex cases.
  • Existing Metrics Fall Short: Standard evaluation metrics like LAS, MLAS, and BLEX show only a weak correlation with compchain classification performance. This implies that a parser can achieve high scores on these metrics without necessarily being good at recovering complex predicate-argument structures.
  • Generalization Gap: Parsers trained on UD data struggle to generalize to structures with deeper embeddings than what's present in their training data, unlike human learners who can acquire complex grammars.
  • Word Order Impact: While word order (VO vs. OV) doesn't significantly impact overall compchain classification, it does show a notable difference for length-2 compchains (RC and RX), with VO languages performing better. This suggests UDPipe parsers might have more difficulty with left-branching structures common in OV languages.
  • Sentence Length Correlation: Compchain classification accuracy shows a weak correlation with sentence length, with OV languages exhibiting a slightly higher (though still modest) correlation.

Key Results

  • English Treebanks: The EWT treebank's baseline parser performed best as a compchain classifier among the English treebanks. However, all three baseline models showed a significant drop in accuracy when evaluated on sentences with predicate-argument structure embedding.
  • Multilingual Evaluation: Performance on compchain classification was only weakly correlated with standard metrics (LAS, MLAS, BLEX) across the 61 UD treebanks.
  • Compchain Length Matters: Parser accuracy dropped precipitously as the length of the compchain (degree of embedding) increased. For example, performance on compchains of length 3 or more was significantly worse than for length 1 or 2.
  • Specific Compchain Performance: For English treebanks, parsers performed better on the 'R' (no embedding) compchain and 'RX' (one level of embedding) than on 'RC', 'RCC', 'RCX', 'RXC', and 'RXX' (more complex embeddings).
  • Word Order and Length: AUC values for compchain classification were generally below 0.7, indicating modest performance. OV languages showed slightly higher AUCs than VO languages for sentence length correlation.

Limitations and Future Work

  • Data Sparsity: The primary limitation identified is the sparsity of deeply embedded predicate-argument structures in current UD test sets. This hinders the evaluation of parsers' ability to generalize to complex structures.
  • Future Work - Data Augmentation: The authors suggest augmenting UD treebanks with more examples of predicate-argument structures with embeddings of degree two or more. This would provide a better testbed for evaluating generalization capabilities, similar to how children acquire complex grammars.
  • Future Work - Parser Development: The findings suggest a need for developing parsers that are more robust to complex syntactic structures and can better recover predicate-argument relationships in the presence of embedding.
  • Future Work - Metric Refinement: While compchains offer a valuable new perspective, further research could explore refining these metrics or developing complementary evaluation methods to capture predicate-argument structure recovery more comprehensively.
  • Limitations of Compchain Metric: The paper acknowledges that compchains are a proxy and don't capture all aspects of predicate-argument structure. Also, the definition of compchains has constraints that might exclude some valid UD parse trees.

Summary updated

BibTeX
@inproceedings{indurkhya2021evaluating,
  author = {Sagar Indurkhya and Beracah Yankama and Robert C. Berwick},
  title = {{Evaluating Universal Dependency Parser Recovery of Predicate Argument Structure via CompChain Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure