TL;DR
This paper investigates how to improve the reliability of labeling word complexity for natural language processing tasks. They found that asking annotators to make comparative judgments (ranking words relative to each other) leads to significantly higher agreement and potentially faster data collection compared to traditional binary classification (complex/not complex).
What Problem Does This Paper Address?
Lexical simplification systems aim to make text easier to understand by replacing complex words with simpler alternatives. A crucial first step in building these systems is identifying which words are complex in a given context. This "Complex Word Identification" (CWI) task is typically performed by human annotators who label words. However, existing CWI datasets suffer from low inter-annotator agreement, meaning human labelers often disagree on what constitutes a complex word. This inconsistency makes it difficult to train reliable machine learning models. The authors hypothesize that the way complexity is presented to annotators (binary vs. comparative) and the homogeneity of the annotator group play a significant role in this agreement.
Methods
The researchers designed a study comparing two annotation interfaces:
- Binary Classification: Annotators were asked to click on words they considered complex (similar to Task 1 in Figure 1).
- Comparative Judgment (Ranking): Annotators were asked to rank highlighted words by their complexity, from most to least complex (similar to Task 2 in Figure 1).
They employed a within-subjects design, where each of the 30 annotators used both interfaces to label 20 sentences each. To address the issue of annotator variability, they specifically selected participants with a homogeneous background: native English speakers, graduate degrees, and within a similar age range (21-30). This controlled approach aimed to isolate the effect of the labeling method on agreement.
Datasets
The study used 20 sentences extracted from the Yimam et al. (2017) dataset. This dataset was chosen for its perceived reliability and previous use in CWI tasks. The sentences were selected to contain words of varying complexity (hard, medium, low), with complexity levels defined based on the proportion of annotators who had previously marked them as complex.
Key Insights and Ideas
- Complexity is a Continuum: The paper argues that word complexity isn't a simple yes/no question but rather a spectrum. Presenting it as such through comparative judgments aligns better with human intuition.
- Homogeneity Matters: The study suggests that controlling for annotator background (language, education, age) can lead to higher agreement, even with traditional binary labeling.
- Comparative Judgments are More Reliable: The core finding is that asking annotators to rank words leads to significantly better agreement than asking them to simply classify words as complex or not.
- Efficiency Gains: Comparative judgments also appear to be faster, suggesting a more efficient way to collect high-quality labels.
- Leveraging Existing Annotations: Comparative rankings can potentially be used to infer additional labels without new annotation effort, by building upon pre-labeled frameworks.
Key Results
- Higher Agreement with Comparative Judgments:
- Cohen's Kappa for the comparative task was 0.6775, which is interpreted as moderate to good agreement.
- Cohen's Kappa for the binary task was 0.3937, indicating minimal agreement.
- Krippendorff's Alpha also showed higher values for the comparative task (0.6821) compared to the binary task.
- Faster Labeling: The average time per sentence for the ranking task was 28.77 seconds, compared to 38.69 seconds for the binary task, a statistically significant difference.
- Improved Agreement over Previous Studies: The agreement scores achieved in this study (even for the binary task) were higher than those reported in previous CWI datasets (e.g., a = 0.244 in Paetzold and Specia, 2016; к = 0.398 in Specia et al., 2012).
Limitations and Future Work
The authors acknowledge several limitations:
- Controlled Experimentation: While they controlled for annotator homogeneity, they did not explicitly control for differences between the two user interfaces or the pre-annotation required for the ranking task.
- Generalizability: The study was conducted with a specific, homogeneous group of annotators. Further research is needed to investigate the effects of annotator group homogeneity more thoroughly and to test these findings with more diverse annotator populations.
- Efficiency Study: The efficiency comparison was not fully controlled, as the ranking task involved pre-selected words. A more thorough experimentation on the comparative efficiency of different approaches is suggested.
Future work includes:
- Investigating the effects of annotator group homogeneity more deeply.
- Conducting a more detailed study on the efficiency of comparative judgments versus binary judgments.
- Exploring the applicability of comparative judgments to other natural language tasks where subjective user experiences (like simplicity) need to be modeled.
Summary updated