Abstract
Word sense annotation is a critical but challenging task in Natural Language Processing (NLP), often resulting in low inter-annotator agreement due to the fine-grained nature of lexicons like WordNet. We present Valido, a visual adjudication tool that utilizes the Structural Semantic Interconnections (SSI) algorithm to assist linguists in resolving these disagreements. By visualizing semantic coherence through graph-based justifications, Valido improves the efficiency and accuracy of the validation process. Evaluation on the SemCor corpus demonstrates that Valido achieves an overall precision of 73.14%, providing a significant boost over random baselines.
Introduction
In NLP, "sense annotation"—the task of assigning a specific dictionary definition to a word based on its context—is notoriously difficult. Even for human experts, popular lexicons like WordNet can be overly fine-grained, making the differences between word senses incredibly subtle. Consequently, inter-annotator agreement often hovers around 70%, necessitating a third-party "validator" to adjudicate disagreements. Manual adjudication is slow and mentally taxing. Valido aims to bridge this gap by providing a tool that offers a visual, structural justification for sense selection, rather than just suggesting a "correct" answer.
Methodology
The core of Valido is the Structural Semantic Interconnections (SSI) algorithm. SSI represents WordNet senses as nodes in a graph, with edges representing relationships such as hypernymy ("is-a-kind-of") or meronymy ("is-part-of"). This graph is further enriched with collocation data from resources like the Oxford Collocations Dictionary.
To determine the most likely sense, SSI identifies valid semantic "paths" between words in a sentence using a Context-Free Grammar (CFG). Each potential sense is assigned a connectivity score based on how many valid paths it forms with other words in the context. The Valido interface (Figure 1) translates these scores into a "traffic-light" system: green for full agreement, red for unresolved disagreement, and orange for cases where the tool suggests a "best fit."
When a user selects a word, Valido displays the semantic graph, showing exactly how that specific sense links to the rest of the sentence. This "virtual annotator" approach allows validators to see the specific lexical chains (e.g., street → way → thoroughfare) that led to a suggestion, making Valido an early example of explainable AI in NLP.
Key Results
The authors evaluated Valido by simulating a disagreement scenario using 1,000 sentences from the SemCor corpus. The tool's suggestions were compared against the dataset's "gold standard" labels. As shown in Figure 2, the tool achieved an overall precision of 73.14%.
Performance varied by part of speech, with nouns achieving the highest precision at 75.80%. While precision remained high across categories, recall was notably lower for adjectives (22.33%) and verbs (43.14%). This limitation stems from the underlying WordNet knowledge base, which often lacks sufficient semantic edges to connect these parts of speech to their surrounding context.
Conclusion
Valido demonstrates that maximizing semantic coherence is a viable metric for resolving annotation disagreements. By providing structural justifications, it empowers human validators to make informed decisions more efficiently. Future work will focus on improving recall by expanding the semantic relationships within the knowledge base and exploring the use of Valido during the initial annotation phase to prevent disagreements before they occur.
Summary updated