Learning to parse with IAA-weighted loss

Héctor Martínez Alonso, Barbara Plank, Arne Skjærholt, Anders Søgaard

Learning to parse with IAA-weighted loss

Introduction

In natural language processing, annotator disagreement is traditionally treated as noise to be filtered out. This research challenges this convention, arguing that disagreement contains valuable linguistic information about inherent ambiguity. By weighting a dependency parser's loss function based on inter-annotator agreement (IAA), the authors achieve significant performance gains across six different languages.

The standard pipeline for creating machine learning datasets involves strict guidelines and an "adjudication" phase where a senior researcher resolves disagreements to create a single "ground truth." However, language is inherently ambiguous. Some sentences genuinely have multiple valid interpretations, and forcing a single "correct" answer hides this complexity from the model. In dependency parsing—the task of determining the grammatical structure of a sentence—this ambiguity is rampant. This paper investigates how to integrate human uncertainty directly into the training process.

Methods

The authors employ a cost-sensitive perceptron classifier within a transition-based dependency parser. The core innovation lies in the calculation of the loss function. Instead of treating every error equally, they introduce a weight, $\gamma$, which is the inverse of the confusion probability estimated from doubly-annotated data.

If annotators consistently agree on a specific label, the model is penalized heavily for missing it. Conversely, if annotators frequently disagree (e.g., confusing a "Subject" with an "Object"), the model is penalized less. This acts as a form of regularization, preventing the model from overfitting to arbitrary choices made during the adjudication of ambiguous cases.

To capture these disagreements, five different factorizations of the tree structure are proposed to estimate confusion probabilities:

  1. LABEL: Disagreement on the dependency label only.
  2. LABELD: Label plus the direction of the head.
  3. CHILDPOSD: The Part-of-Speech (POS) of the dependent word plus direction.
  4. HEADPOS: The POS of the head word.
  5. HEADPOSD: The POS of the head word plus direction.
Visual representation of the five proposed factorizations (a-e) for capturing inter-annotator disagreement.

Datasets

The approach was evaluated across six languages: Norwegian (NO), English (EN), Danish (DA), Catalan (CA), Croatian (HR), and Finnish (FI). Small samples of doubly-annotated data (ranging from 63 to 400 sentences) were used to estimate the confusion matrices for loss weighting, while primary training utilized standard treebanks.

Key Results and Insights

The method demonstrated robust improvements in Labeled Attachment Score (LAS) across five of the six languages. The best-performing setup, Norwegian using the CHILDPOSD factorization, achieved an average overall error reduction of 6.4%.

Relative error reduction across languages and factorizations. Bold values indicate the best factorization for each language.

Key insights from the study include:

  • Disagreement as Signal: Human inconsistencies often reflect systematic linguistic hard cases rather than random error.
  • Selective Under-fitting: By using the inverse of confusion probability, the parser is encouraged to "under-fit" on ambiguous phenomena, allowing for better generalization to new data.
  • Language-Dependent Factorization: The optimal way to measure disagreement varies by language. While LABELD emerged as a consistently robust strategy across the board, specific factorizations like CHILDPOSD or HEADPOS performed better for certain languages.
  • Validation via Shuffling: To ensure improvements were not due to random regularization, the authors tested "shuffled" confusion matrices. Performance dropped, confirming that the linguistic content of human disagreement was the key driver of success.

Conclusion

This work demonstrates that incorporating human uncertainty into model training via IAA-weighted loss is a viable strategy for improving dependency parsing. While the method requires some doubly-annotated data, the required volume is small. Future work may explore automatically determining the optimal factorization for a given dataset and investigating interactions with more complex morphological feature sets.

Summary updated

Abstract

Natural language processing (NLP) annotation projects employ guidelines to maximize inter-annotator agreement (IAA), and models are estimated assuming that there is one single ground truth. However, not all disagreement is noise, and in fact some of it may contain valuable linguistic information. We integrate such information in the training of a cost-sensitive dependency parser. We introduce five different factorizations of IAA and the corresponding loss functions, and evaluate these across six different languages. We obtain robust improvements across the board using a factorization that considers dependency labels and directionality. The best method-dataset combination reaches an average overall error reduction of 6.4% in labeled attachment score.
BibTeX
@inproceedings{alonso2015learning,
  author = {Héctor Martínez Alonso and Barbara Plank and Arne Skjærholt and Anders Søgaard},
  title = {{Learning to parse with IAA-weighted loss}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1152},
}
Figure