Overview
This research presents a cross-lingual effort to annotate regular polysemy—a linguistic phenomenon where words have multiple, predictable, and related meanings. For instance, the word "chicken" can refer to an animal or a type of meat. The authors developed a gold-standard dataset for English, Danish, and Spanish, demonstrating that resolving annotator disagreements using probabilistic modeling (MACE) captures the nuance of "underspecified" meanings more effectively than traditional majority voting.
The Challenge of Semantic Overlap
In traditional Natural Language Processing (NLP), Word Sense Disambiguation (WSD) typically treats word meanings as mutually exclusive: a word is assigned either Sense A or Sense B. However, human language is rarely that rigid. The authors focus on two specific phenomena:
- Regular Polysemy: Predictable shifts in meaning based on semantic classes, known as "dot types." For example, a "Location" (e.g., England the territory) often shifts to an "Organization" (e.g., England the political entity).
- Underspecification: Instances where a word carries both senses simultaneously or where the context does not force a specific choice.
Existing datasets often lack gold-standard labels for these overlapping meanings, making it difficult to train models that can handle the inherent flexibility of human language.
Methodology and Datasets
The researchers selected 500 examples for several "dot types" across three languages. The English data was sourced via Amazon Mechanical Turk (AMT), while Danish and Spanish data relied on native-speaker volunteers. To determine the "correct" sense from multiple human annotations, they employed two strategies:
- Majority Voting with Theory-Compliant Backoff: A hierarchical approach where ties were resolved based on theoretical assumptions (e.g., defaulting to "Literal" over "Underspecified").
- MACE (Multi-Annotator Competence Estimation): An unsupervised system using Expectation-Maximization to estimate annotator reliability and predict the most likely "true" label without relying on theoretical hierarchies.
Key Insights: The Impact of Semantic Type
A primary finding was that annotator agreement varied significantly based on the semantic class (dot type) rather than just the language. For example, "Animal/Meat" was relatively easy for humans to agree on, whereas "Process/Result" (e.g., "the building took months" vs. "the building is sturdy") proved nearly impossible to reconcile.
Furthermore, the study revealed a "Turker bias": paid crowdsourced workers were significantly less likely to choose the "Underspecified" option compared to volunteers. They tended to force a binary choice, likely because identifying underspecification requires a higher level of cognitive abstraction.
Results and Distribution Shifts
The use of MACE acted as a "smoother," generally increasing the frequency of the "Underspecified" tag compared to majority voting. In the English "Artifact/Information" category (e.g., a book), majority voting identified 54 underspecified cases, while MACE identified 150. This suggests that probabilistic models are much more sensitive to semantic overlap than simple voting mechanisms.
Despite differences in corpora and annotator types, the behavior of specific dot types remained remarkably consistent across English, Danish, and Spanish. The "Location/Organization" type showed high agreement across all three languages, suggesting it is a robust semantic category for future NLP tasks.
Conclusion and Future Work
While this study focuses exclusively on nouns, it provides a vital gold-standard dataset for training automated WSD systems. Future work will explore whether machine learning models can learn to predict these complex "underspecified" tags as accurately as humans or probabilistic models like MACE. The authors also highlight the need for caution when using low-cost crowdsourcing for high-level semantic tasks due to the observed bias against complex labels.
Summary updated