Learning to parse with IAA-weighted loss
Introduction
In natural language processing, annotator disagreement is traditionally treated as noise to be filtered out. This research challenges this convention, arguing that disagreement contains valuable linguistic information about inherent ambiguity. By weighting a dependency parser's loss function based on inter-annotator agreement (IAA), the authors achieve significant performance gains across six different languages.
The standard pipeline for creating machine learning datasets involves strict guidelines and an "adjudication" phase where a senior researcher resolves disagreements to create a single "ground truth." However, language is inherently ambiguous. Some sentences genuinely have multiple valid interpretations, and forcing a single "correct" answer hides this complexity from the model. In dependency parsing—the task of determining the grammatical structure of a sentence—this ambiguity is rampant. This paper investigates how to integrate human uncertainty directly into the training process.
Methods
The authors employ a cost-sensitive perceptron classifier within a transition-based dependency parser. The core innovation lies in the calculation of the loss function. Instead of treating every error equally, they introduce a weight, $\gamma$, which is the inverse of the confusion probability estimated from doubly-annotated data.
If annotators consistently agree on a specific label, the model is penalized heavily for missing it. Conversely, if annotators frequently disagree (e.g., confusing a "Subject" with an "Object"), the model is penalized less. This acts as a form of regularization, preventing the model from overfitting to arbitrary choices made during the adjudication of ambiguous cases.
To capture these disagreements, five different factorizations of the tree structure are proposed to estimate confusion probabilities:
- LABEL: Disagreement on the dependency label only.
- LABELD: Label plus the direction of the head.
- CHILDPOSD: The Part-of-Speech (POS) of the dependent word plus direction.
- HEADPOS: The POS of the head word.
- HEADPOSD: The POS of the head word plus direction.
Datasets
The approach was evaluated across six languages: Norwegian (NO), English (EN), Danish (DA), Catalan (CA), Croatian (HR), and Finnish (FI). Small samples of doubly-annotated data (ranging from 63 to 400 sentences) were used to estimate the confusion matrices for loss weighting, while primary training utilized standard treebanks.
Key Results and Insights
The method demonstrated robust improvements in Labeled Attachment Score (LAS) across five of the six languages. The best-performing setup, Norwegian using the CHILDPOSD factorization, achieved an average overall error reduction of 6.4%.
Key insights from the study include:
- Disagreement as Signal: Human inconsistencies often reflect systematic linguistic hard cases rather than random error.
- Selective Under-fitting: By using the inverse of confusion probability, the parser is encouraged to "under-fit" on ambiguous phenomena, allowing for better generalization to new data.
- Language-Dependent Factorization: The optimal way to measure disagreement varies by language. While LABELD emerged as a consistently robust strategy across the board, specific factorizations like CHILDPOSD or HEADPOS performed better for certain languages.
- Validation via Shuffling: To ensure improvements were not due to random regularization, the authors tested "shuffled" confusion matrices. Performance dropped, confirming that the linguistic content of human disagreement was the key driver of success.
Conclusion
This work demonstrates that incorporating human uncertainty into model training via IAA-weighted loss is a viable strategy for improving dependency parsing. While the method requires some doubly-annotated data, the required volume is small. Future work may explore automatically determining the optimal factorization for a given dataset and investigating interactions with more complex morphological feature sets.
Summary updated