Sprucing up the trees – Error detection in treebanks

Ines Rehbein, Josef RuppenhoferCOLING 2018

TL;DR

This paper introduces MACE-AL-TREE, a novel method for automatically detecting and correcting errors in dependency parse trees. By combining ensemble parsing, Bayesian inference, and active learning, it significantly improves the quality of both manually and automatically annotated treebanks, offering a valuable tool for data cleaning and domain adaptation in NLP.

What Problem Does This Paper Address?

Syntactic parsing is crucial for many Natural Language Processing (NLP) applications, and dependency parse trees are a popular format for representing syntactic structure. However, creating high-quality, manually annotated treebanks is a laborious and expensive process. Existing treebanks, especially those automatically converted from other formats or created with limited funding, often contain annotation noise and errors. While manual quality checks are ideal, they are often infeasible. This paper addresses the critical need for automated methods to detect and correct these errors, which is essential for improving the reliability of NLP models and for tasks like domain adaptation and working with low-resource languages.

Methods

The core of the proposed method is MACE-AL-TREE, an extension of the MACE-AL framework. MACE-AL itself combines variational inference (VI) with active learning (AL) to estimate annotator reliability. For tree structures, MACE-AL-TREE tackles two main challenges:

  • Error Encoding: It distinguishes between two types of errors:
  • Labeling errors: The wrong label is assigned to a directed arc (e.g., "nsubj" instead of "obj").
  • Attachment errors: The relation between two nodes is incorrectly predicted (i.e., the wrong head is assigned). To handle these, two separate variational models are trained: one for edge labels and one for attachments.

  • Tree Generation: After identifying potential errors and obtaining corrected labels/attachments, the system needs to reconstruct coherent trees. Two methods are proposed:

  • MACE-AL-TREE: Directly uses the final predictions from the VI models to construct trees. This might not always result in fully connected trees.
  • MACE-AL-TREE-CLE: Employs the Chu-Liu-Edmonds (CLE) algorithm to select the highest-scoring, well-formed tree from a weight matrix derived from parser committee votes and competence estimates.

The overall workflow involves preprocessing parser outputs from a committee of parsers, training the VI models, using active learning to select instances for human correction (or simulated correction from gold trees), and then iteratively retraining the models. A key advantage is that the parsers themselves are only trained once for preprocessing, avoiding costly retraining during active learning.

Datasets

The experiments were conducted on:

  • German UD treebank: Used for error detection in manually annotated trees.
  • English web treebank (LDC2012T13): Originally annotated with constituency trees and converted to Universal Dependencies. This dataset, comprising five genres (reviews, weblogs, answers, emails, newsgroups), was used for out-of-domain error detection and domain adaptation studies.
  • German TiGer treebank: Used for in-domain error detection on German newspaper text.

Key Insights and Ideas

  • Separating Label and Attachment Errors is Crucial: Treating dependency labels and head attachments as distinct error types allows for more targeted and effective error detection and correction in tree structures.
  • Active Learning Significantly Boosts Quality: Even with minimal human intervention, active learning guided by the model's uncertainty (high posterior entropy) effectively identifies errors, leading to substantial improvements in treebank quality.
  • Ensemble Parsing is Key: Leveraging predictions from multiple parsers (an ensemble) provides a more robust signal for error detection than relying on a single parser. The diversity of parsers is highlighted as more important than individual parser complexity.
  • Generative Model Adds Value: The unsupervised generative model (MACE) plays a crucial role in estimating annotator (parser) reliability, which is then used to guide the active learning process and improve overall accuracy.
  • Out-of-Domain Performance is Promising: The method demonstrates effectiveness not only on in-domain data but also on out-of-domain data, suggesting its utility for adapting models to new text genres or domains.

Key Results

  • Error Detection Precision: The method achieved error detection precision ranging from 35% to over 90% across different genres and scenarios.
  • LAS Improvement:
    • In out-of-domain settings (English web treebank), MACE-AL-TREE achieved LAS increases of 1-2% over simpler ensemble methods after 1,000 iterations of active learning.
  • On the German TiGer treebank (in-domain), LAS increased from 88.8% to 94.6% after 1,000 iterations of active learning, with a reasonable number of corrections.
  • The method also showed gains in accuracy for domain adaptation with self-training, improving LAS by up to 2% on some genres.
  • Efficiency: The approach improves treebank quality without requiring retraining of the parsers themselves, making it computationally more efficient than methods that necessitate parser retraining.

Limitations and Future Work

  • Human Annotation Effort: While active learning minimizes effort, some human annotation is still required for manual correction. The paper notes that the simulation study using gold trees allows for systematic evaluation without human annotators.
  • Potential for Non-Connected Trees: The basic MACE-AL-TREE method might produce non-connected trees, though the CLE algorithm mitigates this.
  • Review Subcorpus Anomaly: The review subcorpus showed some unusual behavior in error detection precision, which the authors attribute to potential idiosyncrasies or inconsistencies in the gold annotations.
  • Future Work: The authors suggest exploring the impact of more unlabelled data for self-training, as indicated by previous research, and further investigating the performance on genres with a higher proportion of unknown words. They also mention that their method can be applied to improve parsing accuracy on out-of-domain data without human feedback.

Summary updated

BibTeX
@inproceedings{rehbein2018sprucing,
  author = {Ines Rehbein and Josef Ruppenhofer},
  title = {{Sprucing up the trees – Error detection in treebanks}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure