TL;DR
This paper introduces "syntagmatic embeddings," a novel unsupervised method for learning selectional preferences (SP) by leveraging directional word context windows. Their approach demonstrates that by encoding word order, they can achieve SP learning comparable to supervised methods, even without relying on dependency-parsed corpora.
What Problem Does This Paper Address?
Selectional Preference (SP) describes how words tend to semantically select other words they co-occur with in specific syntactic relationships (e.g., which adjectives typically modify a noun, or which objects a verb typically takes). This is crucial for understanding language and for various Natural Language Processing (NLP) tasks like word sense disambiguation. Traditionally, learning SP has been a supervised task requiring dependency-parsed corpora, which are expensive to create and not available for many languages. While unsupervised methods like Word2vec and GloVe capture some SP, they are generally outperformed by supervised methods because they don't explicitly model syntactic relationships. This paper aims to bridge this gap by developing an unsupervised method that effectively captures SP without the need for parsed data.
Methods
The core innovation is the concept of syntagmatic embeddings. Unlike traditional word embeddings (like Word2vec or GloVe) that use symmetric context windows (looking equally to the left and right of a word), this method uses directional context windows. This means a word gets two distinct embeddings: one for its left context and one for its right context.
The authors achieve this by:
- Directional Co-occurrence Matrix: They build a co-occurrence matrix where the entry
M_ijrepresents how many times wordiappeared to the left of wordjwithin a defined window size. The entryM_jithen represents how many times wordjappeared to the left of wordi. - Truncated Singular Value Decomposition (SVD): They apply SVD to this directional co-occurrence matrix to reduce its dimensionality and noise, obtaining low-rank factor matrices. These matrices represent the left and right context embeddings for each word.
- Exponential Weighting: They scale the singular vectors by an exponential factor of their singular values, allowing for fine-tuning of the embeddings for different tasks.
- Weighted Factorization (inspired by GloVe): They also explore a weighted factorization model, similar to GloVe, to see if weighting co-occurrence terms based on their frequency improves performance.
Datasets
- British National Corpus (BNC): Used for training the word embeddings. The corpus was preprocessed by normalizing casing, removing punctuation, and filtering words that appeared less than 100 times.
- SP-10K: Used for evaluating the learned selectional preferences. This dataset contains human-annotated plausibility scores for word pairs in specific syntactic relations (amod, nsubj, dobj). The authors focus on the direct relations within SP-10K.
Key Insights and Ideas
- Word Order as a Proxy for Syntax: The paper demonstrates that by explicitly modeling word order (left vs. right context), one can effectively capture syntactic relationships and thus selectional preferences, even without explicit syntactic parsing.
- Directional Embeddings Outperform Symmetric Ones: Syntagmatic embeddings, with their directional nature, consistently outperform paradigmatic embeddings (like standard Word2vec and GloVe) on SP tasks.
- Unsupervised Learning Can Be as Good as Supervised: Surprisingly, their unsupervised syntagmatic model (spvec) achieves performance comparable to, and in some cases better than, supervised models that rely on dependency parses. This suggests that dependency parsing might not always be necessary for good SP learning.
- Parameter Sensitivity: The performance of the models is sensitive to parameter choices, with term-weighting (specifically
log) and dimensionality being particularly important. The exponential weighting factorpalso plays a significant role, withp=0.5showing good results. - Weighted Factorization Can Be Detrimental for SP: While weighted factorization (like in GloVe) is beneficial for tasks like word similarity, it appears to be less effective or even detrimental for learning selectional preferences compared to unweighted SVD factorization in this context.
Key Results
- Superiority over Paradigmatic Models: In 69% of parameter combinations, the syntagmatic model showed improvement over paradigmatic models, with an average correlation increase of 0.14 points (a 54% improvement). The maximum correlation achieved by a syntagmatic model was 0.71, compared to 0.58 for the best paradigmatic model.
- Competitive with Supervised Models: The unsupervised syntagmatic model (spvec) achieved a Spearman's correlation of 0.714 on the SP-10K dataset, which was comparable to supervised models like
spvec-s(0.712) andspvec-sr(0.717). - Parameter Impact: Term-weighting, particularly using the
logfunction, was found to be the most impactful parameter, yielding an average correlation of 0.55 ± 0.07. Embedding dimensionality was the next most significant factor. - Window Size Preferences: Different dependency relations showed varying preferences for window size in the unsupervised model (e.g.,
amodpreferred window size 1,dobjpreferred 4), suggesting that optimal window sizes can be relation-specific.
Limitations and Future Work
- Limitations:
- The authors acknowledge that while their unsupervised model is competitive, supervised models still show a slight advantage for
amodanddobjrelations, suggesting that dependency information can still be beneficial.
- The authors acknowledge that while their unsupervised model is competitive, supervised models still show a slight advantage for
- The paper primarily focuses on English, and its applicability to other word-ordered languages needs further investigation.
- The computational cost of parsing large corpora for supervised methods is mentioned as a drawback, but the performance gains might not always justify the compute.
- Future Work:
- The authors suggest that their syntagmatic embeddings could be a valuable resource for resource-poor languages.
- They propose further exploration of structural biases like directionality in simpler models for other NLP tasks.
- Investigating the impact of different term-weighting schemes and exponential weighting factors for various tasks is also suggested.
- The paper hints at the potential for word order to contribute to semantics even when not directly related by a dependency, which could be a fruitful area for future research.
Summary updated