TL;DR
This paper tackles the challenge of identifying word senses that are not present in a given sense inventory, a common issue in natural language processing. They frame this as an "outlier detection" problem, treating known senses as "normal" data and unknown senses as anomalies. Their novel approach, combining a nearest-neighbor distance metric with a method to combat data sparseness by sharing training data across related lemmas, achieves promising results for practical applications.
What Problem Does This Paper Address?
In natural language processing, systems often rely on predefined sets of word senses (sense inventories) to understand text. However, these inventories are rarely exhaustive. They might miss domain-specific senses or simply be incomplete due to the vastness of language. When a system encounters a word usage that doesn't fit any of its known senses, it can lead to incorrect analysis, impacting downstream tasks like inference or semantic parsing. This paper addresses the problem of "unknown word sense detection" – how to reliably identify these out-of-inventory word senses.
Methods
The core of the paper's contribution lies in framing unknown word sense detection as an outlier detection problem. They model "normal" data as occurrences of known word senses from a sense-annotated corpus. Any occurrence that deviates significantly from this "normal" data is flagged as an unknown sense.
Their primary method is a Nearest Neighbor (NN)-based outlier detection approach. This technique works by comparing the distance of a new data point (a word occurrence) to its nearest neighbor in the training set with the distance between that nearest neighbor and its nearest neighbor. If the new data point is much farther from its nearest neighbor than that neighbor is from its own, it's considered an outlier (an unknown sense).
To address the issue of data sparseness (where there might not be enough examples for certain senses or lemmas), they introduce a crucial enhancement: sharing training data across lemmas. This means that for a given lemma, they also use annotated data from other lemmas that belong to the same semantic frame in FrameNet. This effectively increases the amount of "normal" data available for comparison, making the system more robust.
Datasets
The primary dataset used in this paper is FrameNet 1.2. FrameNet is a lexical resource that describes semantic frames and links them to words and expressions that evoke them, providing sense annotations and semantic roles. The authors specifically focus on ambiguous lemmas within FrameNet for their experiments.
Key Insights and Ideas
- Unknown Sense Detection as Outlier Detection: This is the central conceptual shift. By reframing the problem, they can leverage established outlier detection techniques, which are designed to find anomalies rather than classify into predefined categories.
- Data Sparseness is a Major Hurdle: The paper repeatedly highlights how limited training data for specific senses or lemmas significantly impacts performance, particularly recall.
- Sharing Data Across Related Lemmas is Crucial: The most significant performance gains come from augmenting training data by including examples from lemmas within the same semantic frame. This demonstrates the power of leveraging semantic relationships to overcome data scarcity.
- Contextual Features are Key: Features derived from the surrounding words (bag-of-words, lemmas, POS, NE) prove to be more effective for unknown sense detection than purely syntactic features, especially when combined with the data augmentation strategy.
Key Results
The paper presents a progression of experiments, showing increasing improvements:
- Experiment 1 (WSD Confidence Scores): Using confidence scores from a standard Word Sense Disambiguation (WSD) system as a proxy for unknown senses yielded very low recall (around 0.17 at a high threshold), indicating this simple approach is insufficient.
- Experiment 2 (NN-based Outlier Detection): The NN-based outlier detection approach showed a marked improvement in recall compared to Experiment 1, reaching up to 0.35 with certain feature sets. However, recall was still considered too low for practical use.
- Experiment 3 (NN-based Outlier Detection with Added Training Data): This experiment, which incorporated the data sharing strategy, achieved the best results. With context-based features (All and Cx), the system reached 0.77 precision and 0.82 recall on FrameNet 1.2. These results are deemed "good enough for the system to be usable in practice."
Limitations and Future Work
- Limitations:
- The paper acknowledges that even with data augmentation, performance is still sensitive to the uniformity of the training set (i.e., lemmas with more senses are harder to handle).
- The current NN approach considers only the single nearest neighbor, which might not be precise enough for boundary cases.
- While context features are effective, the paper suggests exploring methods for further generalization over context words to combat data sparseness.
- Future Work:
- Investigating the use of more than one nearest neighbor in the outlier detection to compute more precise boundaries.
- Exploring more sophisticated feature vectors that generalize better over context words.
- Testing other outlier detection approaches beyond the nearest neighbor method.
- Integrating unknown sense detection into WSD systems to filter out ambiguous items that the WSD system cannot handle, and then potentially assigning these unknown senses to similar existing senses or clustering them.
Summary updated