TL;DR
This paper introduces a novel augmentation to Cohen's kappa statistic to better evaluate interannotator reliability when annotators can assign multiple labels to data points. This method provides a more nuanced understanding of agreement, especially when secondary labels are involved, and helps identify problematic data points for annotation.
What Problem Does This Paper Address?
In many natural language processing tasks, such as text classification and summarization, reliable annotated data is crucial for training and evaluating machine learning models. A common way to measure the consistency of human annotators is using Cohen's kappa statistic, which compares observed agreement to agreement expected by chance. However, the standard kappa statistic is designed for situations where each data point receives only a single label. This paper addresses the challenge of evaluating interannotator reliability when annotators are allowed to assign multiple labels to a single data point, a scenario that arises in projects like classifying email messages where a message might be both a "question" and an "answer." The standard approach of simply ignoring secondary labels is problematic because it discards valuable information and misrepresents the annotation process.
Methods
The authors propose an augmentation to Cohen's kappa statistic. Instead of treating each label assignment as a binary (present/absent) choice, they introduce a weighting system. When an annotator assigns a single label, it receives a weight of 1.0. When two labels are assigned (primary and secondary), the primary label receives a weight of p and the secondary label receives a weight of (1-p), where p is a value between 0.5 and 1.0. By varying p, the authors can explore different levels of agreement. A p of 1.0 effectively ignores secondary labels, while a p of 0.5 gives equal weight to primary and secondary labels. This weighted approach allows for the calculation of a more nuanced agreement score, particularly for partial agreement scenarios (e.g., one annotator labels a message as {primary: question, secondary: answer} and another as {primary: answer, secondary: question}). The paper details how to compute the observed agreement (p(A)) and expected agreement (p(E)) using these weighted annotations, and then applies the standard kappa formula with these new values to calculate an augmented kappa score (K').
Datasets
The study uses a corpus of 380 email messages collected from members of the Columbia University chapter of the Association for Computing Machinery (ACM) in 2001. These messages were annotated by two annotators with one or two labels from a set of ten categories. This corpus is a subset of a larger corpus, with the rest of the larger corpus still undergoing annotation.
Key Insights and Ideas
- Beyond Binary Agreement: The core insight is that interannotator agreement is not always binary. When multiple labels are allowed, partial agreement is common and can be meaningfully quantified.
- The Role of Secondary Labels: The proposed method allows researchers to investigate how annotators use secondary labels. By varying the weight p, one can see how sensitive the overall agreement is to the inclusion and weighting of secondary information.
- Identifying Annotation Issues: The augmented kappa statistic can be calculated for individual messages at different p values. This allows for the categorization of messages based on their agreement patterns, revealing:
- Messages with consistently high agreement (good examples).
- Messages with consistently low agreement (problematic, requiring annotation revision).
- Messages where agreement drops significantly when secondary labels are considered (suggesting issues with secondary label guidelines).
- Messages where agreement is higher when secondary labels are considered, but primary/secondary roles are sometimes swapped (indicating concurrent features).
- Data-Driven Annotation Improvement: The insights gained from analyzing agreement patterns can directly inform revisions to annotation guidelines, leading to more consistent and reliable datasets.
Key Results
The paper reports that, on average, the augmented kappa statistic did not inflate scores. When secondary labels were ignored (p=1.0), the average kappa was 0.299. When primary and secondary labels were given equal weight (p=0.5), the average kappa was 0.281. This suggests that the secondary labels, as used in this project, did not consistently increase overall agreement. However, the analysis of messages at different p values revealed that:
- 92 out of 380 messages showed consistent high agreement.
- 183 out of 380 messages showed consistently low agreement.
- A significant portion of messages (47/380) showed higher agreement when secondary labels were considered, but with primary/secondary roles sometimes swapped, indicating concurrent features.
Limitations and Future Work
The authors acknowledge that the annotation guidelines for their project may need revision, as evidenced by the inconsistent use of secondary labels and the identification of problematic message categories. They suggest that a strict specification of when secondary labels should be used, and potentially a reconsideration of the ten available labels, could improve annotation quality.
Future work proposed includes developing a "sharpening" algorithm. This algorithm would iteratively refine annotator and message weights based on their agreement scores, aiming to minimize the influence of unreliable annotators and poorly annotated messages, thereby increasing confidence in the final label assignments. This could involve ranking annotators by their agreement with the group and ranking messages by how well the group agrees on their labels.
Summary updated