Learning to parse with IAA-weighted loss

Héctor Martínez Alonso, Barbara Plank, Arne Skjærholt, Anders Søgaard

Abstract

Natural language processing (NLP) annotation projects employ guidelines to maximize inter-annotator agreement (IAA), and models are estimated assuming that there is one single ground truth. However, not all disagreement is noise, and in fact some of it may contain valuable linguistic information. We integrate such information in the training of a cost-sensitive dependency parser. We introduce five different factorizations of IAA and the corresponding loss functions, and evaluate these across six different languages. We obtain robust improvements across the board using a factorization that considers dependency labels and directionality. The best method-dataset combination reaches an average overall error reduction of 6.4% in labeled attachment score.
BibTeX
@inproceedings{alonso2015learning,
  author = {Héctor Martínez Alonso and Barbara Plank and Arne Skjærholt and Anders Søgaard},
  title = {{Learning to parse with IAA-weighted loss}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1152},
}

Loading related content...

Figure