TL;DR
This paper systematically investigates the impact of various parameters on the performance of semantic vector space models (VSMs) for word similarity tasks. The authors provide concrete recommendations for optimal parameter settings, highlighting that smaller vector sizes and specific weighting schemes like Positive Mutual Information (PosMI) and T-Test, combined with the Correlation similarity metric, generally yield the best results.
What Problem Does This Paper Address?
Vector space models (VSMs) are a cornerstone of modern Natural Language Processing, representing words as dense vectors in a high-dimensional space. The core idea is that words appearing in similar contexts tend to have similar meanings (the distributional hypothesis). However, constructing effective VSMs involves a multitude of choices: how large should the vectors be? How do we define a word's "context" (e.g., surrounding words in a window, or syntactic relationships)? What similarity metric should be used to compare these vectors? What weighting schemes best capture the importance of different contexts?
Despite the widespread use of VSMs, there has been a lack of systematic investigation into how these various parameters interact and influence performance across different tasks and corpora. This has led to a fragmented landscape where researchers often choose parameters based on convention or limited prior studies, without a clear understanding of their optimal settings. This paper aims to fill this gap by conducting a comprehensive empirical study to identify the most effective parameter configurations for VSMs.
Methods
The authors employ a rigorous experimental methodology to evaluate the impact of several key parameters on VSM performance. They systematically vary each parameter while keeping others fixed, or by employing a strategy of fixing parameters based on earlier findings.
The core parameters investigated include:
- Vector Size: The dimensionality of the word vectors.
- Window Size: The number of words surrounding a target word considered as its context.
- Context Type: Window-based (co-occurrence within a fixed window) vs. dependency-based (co-occurrence based on syntactic relations).
- Feature Granularity: The level of detail in representing context words (e.g., raw words, stemmed, lemmatized, or POS-tagged).
- Similarity Metric: Various mathematical functions to quantify the similarity between two vectors (e.g., Cosine, Correlation, Dice).
- Weighting Scheme: Methods to assign importance to context words based on their frequency or informativeness (e.g., TF-IDF, Positive Mutual Information, T-Test).
- Stopwords and High Frequency Cut-off: Strategies for handling common words.
They also consider "superparameters" like the choice of evaluation dataset and source corpus. The experiments involve generating VSMs using different parameter combinations and evaluating their performance on various semantic similarity tasks.
Datasets
The study utilizes a range of established datasets for evaluating semantic similarity and compositionality:
- Word Similarity Datasets:
- Rubenstein & Goodenough (RG)
- Miller & Charles (MC)
- WordSim353 (W353)
- MEN dataset
- TOEFL synonym dataset (multiple-choice questions)
- Compositionality Dataset:
- Mitchell and Lapata (2010) dataset for evaluating phrasal similarity.
The source corpora used to build the VSMs include:
- British National Corpus (BNC)
- ukWaC (a large web-crawled corpus)
- Sub-spaced Wikipedia corpus
- Google Syntactic N-gram corpus
Key Insights and Ideas
- Vector Size Plateau: Performance generally plateaus after a certain vector size (around 50,000 features), and increasing it further can even be detrimental. This suggests that adding very rare features doesn't necessarily improve semantic representation and can increase computational cost.
- Window Size vs. Corpus Size: For large corpora, smaller window sizes tend to perform better. However, a large window size can compensate for a smaller corpus, and a sub-spaced corpus can benefit from larger window sizes due to its focused nature.
- Corpus Size Dominance: The size of the source corpus appears to be more critical than the choice between window-based or dependency-based context methods when using similar window sizes.
- Stemming is Key for Granularity: Among feature granularity options, stemming consistently yields the best results, often outperforming lemmatization or more complex tagging schemes. Increasing granularity beyond stemming does not seem to improve performance.
- Weighting Schemes Matter: Positive Mutual Information (PosMI) and T-Test emerge as the top-performing weighting schemes. Their effectiveness can depend on the corpus and vector size.
- Correlation is King for Similarity: The Correlation similarity metric consistently outperforms others, including Cosine, across various settings.
- Weighting Obviates Stopword Removal: Using effective weighting schemes can make the explicit removal of stopwords and high-frequency words less critical, as their weights naturally become low.
- Compositionality Parameter Transfer: Optimal parameters for individual word vectors generally transfer well to phrasal compositionality tasks.
Key Results
- Optimal Vector Size: Around 50,000 features is found to be optimal for most tasks.
- Best Performing Combination: The combination of Positive Mutual Information (PosMI) weighting with the Correlation similarity metric consistently yields high scores across datasets. T-Test is also a strong contender, especially for smaller vector sizes.
- Window Size Impact: For large corpora like ukWaC, a window size of 3 often achieves the best performance.
- Corpus Performance: The ukWaC corpus generally outperforms the BNC across various settings.
- Syntactic N-grams: The Google Syntactic N-gram corpus, despite using smaller vectors, shows very good performance, suggesting its large size and structured nature are beneficial.
- Compositionality: Addition outperforms multiplication for combining vectors into phrases, and small window sizes are generally preferred.
Limitations and Future Work
- Dimensionality Reduction: The study explicitly excluded dimensionality reduction techniques (like PCA or SVD) as they are applied after an initial VSM is constructed and depend on its quality. This remains an area for future exploration.
- Dependency-Based Granularity: The paper notes that further exploration of feature granularity for dependency-based methods is warranted.
- Corpus Characteristics: The relationship between corpus size, frequency distribution, and optimal feature cut-off points is an area for further investigation.
- Online Settings: The paper briefly mentions scenarios where weighting might be computationally expensive or not feasible (e.g., online settings), suggesting further research into these specific contexts.
Summary updated