Abstract
This paper investigates distributional similarity as a solution to the sparse data problem in Natural Language Processing (NLP). By using similar words to estimate the probabilities of unseen co-occurrences, the author compares various metrics and introduces the $\alpha$-skew divergence. Results on a pseudo-word disambiguation task show that similarity-based methods significantly outperform standard backoff techniques, with the $\alpha$-skew divergence achieving state-of-the-art performance by effectively modeling the asymmetric nature of linguistic similarity.
Introduction: The Sparse Data Challenge
In NLP, statistical models often rely on word co-occurrence probabilities. However, language follows a "long tail" distribution: even in massive datasets, many valid word pairs (e.g., "eat an apricot") may never appear. Standard models often mistakenly assign these "unseen events" a probability of zero.
Traditional solutions, such as Katz’s backoff, handle this by falling back to a general average (e.g., the unigram frequency of "apricot"). However, this ignores the specific context of the preceding word. This research proposes using "similar" words as proxies. If "eat an apricot" is unseen but "eat a peach" is frequent, the data for "peach" can be used to estimate the probability for "apricot."
Methodology: Measuring Distributional Similarity
The core approach is distance-weighted averaging, where the probability of an unseen pair $(n, v)$ is estimated by aggregating information from words $m$ that are similar to $n$. The paper evaluates seven well-known similarity measures across vector-based, combinatorial, and information-theoretic categories.
The primary contribution is the $\alpha$-skew divergence, an asymmetric version of the Kullback-Leibler (KL) divergence. KL divergence is typically undefined if a word appears in one distribution but not the other (due to division by zero). The skew divergence solves this by "mixing" a small amount of the target distribution into the source, effectively smoothing the calculation:
$$s_\alpha(q, r) = D(r \parallel \alpha \cdot q + (1 - \alpha) \cdot r)$$
Experimental Setup
The study utilized the 1988 Associated Press newswire, focusing on the 1000 most frequent nouns and their co-occurrences with transitive verbs. The evaluation task was "pseudo-word" disambiguation: for an unseen pair $(n, v_1)$, the model is presented with a triple $(n, v_1, v_2)$ and must decide which verb was the original partner. This forces the model to rely on distributional similarity rather than simple frequency.
Results and Analysis
All similarity-based methods significantly outperformed the standard Katz backoff baseline. While backoff yielded an error rate of 51% (comparable to random guessing on this task), similarity measures reduced the error rate to the 27%–38% range.
The $\alpha$-skew divergence (with $\alpha = 0.99$) achieved the best overall performance, providing a 4% error rate reduction over the Jensen-Shannon divergence, the previous state-of-the-art. Interestingly, the simple Jaccard’s coefficient performed remarkably well despite ignoring probability values, suggesting that the "intersection of supports" (the set of shared contexts) is a critical factor in similarity.
Conclusion and Future Directions
The success of the $\alpha$-skew divergence highlights that similarity in language is often asymmetric; one word may be a good proxy for another without the reverse being true. Future work includes testing these measures on larger-scale tasks like speech recognition and exploring whether the optimal value for $\alpha$ varies based on data sparsity.
Summary updated