Abstract
Standard Word2Vec models, such as Skip-gram and CBOW, are inherently "order-insensitive," treating context as a bag of words. While effective for capturing semantic similarity, they often struggle with syntax-heavy tasks where word order is paramount. This paper introduces two straightforward modifications—Structured Skip-gram and CWindow—that integrate positional information into the training process. These adaptations significantly improve performance on tasks like part-of-speech tagging and dependency parsing while maintaining the computational efficiency of the original architectures.
Introduction
Word embeddings serve as the foundational representation for most modern Natural Language Processing (NLP) systems. Among the most popular tools for generating these vectors is Word2Vec. However, standard Word2Vec models possess a significant limitation: they are designed to capture semantics (meaning) rather than syntax (grammatical function).
Because these models discard word order, they learn that "cat" and "dog" are similar because they appear in similar contexts. However, for syntactic analysis, the exact position of a word relative to its neighbors is critical. For instance, the word "the" is almost always followed by a noun. Standard Word2Vec recognizes that "the" and "cat" co-occur but fails to explicitly model the directional relationship. This makes standard embeddings suboptimal for tasks like identifying parts of speech or mapping structural dependencies.
Methodology
The authors propose two "structured" versions of the classic Word2Vec architectures to bridge this gap.
Baseline Architectures
The original Word2Vec suite consists of the Continuous Bag-of-Words (CBOW) and Skip-gram models. CBOW predicts a target word from its context, while Skip-gram predicts context words from a target word.
Proposed Structured Models
To incorporate order, the authors modify these baselines to create models that are sensitive to "what goes where" rather than just "what goes together."
- Structured Skip-gram: In the original Skip-gram, a single weight matrix predicts all context words. The structured version uses a distinct weight matrix for each relative position (e.g., separate matrices for positions -2, -1, +1, and +2). This forces the model to learn position-specific word distributions.
- Continuous Window (CWindow): While CBOW sums or averages context embeddings, CWindow concatenates them in their specific order. This preservation of sequence makes the model sensitive to the precise arrangement of surrounding words.
Despite the increase in parameters—proportional to the window size—these models remain efficient enough to train on billions of words in less than 24 hours.
Experimental Setup
The models were trained on a large English Wikipedia dump (1.89 billion words) and an English Twitter collection (847 million words). Evaluation focused on two primary syntactic tasks:
- Part-of-Speech (POS) Tagging: Using the English Penn Treebank (PTB) and the ARK Twitter dataset.
- Dependency Parsing: Using the English PTB with Stanford Dependencies.
The results were compared against standard Word2Vec and the well-known SENNA embeddings.
Results and Discussion
The structured models consistently outperformed their unstructured counterparts across all benchmarks.
Quantitative Performance
- POS Tagging: On the Penn Treebank, Structured Skip-gram achieved 97.05% accuracy, surpassing standard Skip-gram (96.68%) and SENNA (96.58%).
- Dependency Parsing: The models showed significant gains in Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS). On PTB, Structured Skip-gram reached 92.24 UAS, compared to 91.52 for standard CBOW.
- Robustness: Improvements were consistent across both formal (Wikipedia) and informal (Twitter) text, suggesting that order-awareness is a universal benefit for syntactic modeling.
Qualitative Insights
Qualitative analysis revealed that the new models group words by functional role. For example, while standard Word2Vec might group "breaking" with "broken" (semantic), CWindow groups "breaking" with "putting" or "turning" (syntactic gerunds).
Conclusion
By making Word2Vec aware of word order, the authors successfully bridge the gap between simple vector models and complex linguistic structures. While these models introduce some computational overhead—CWindow is roughly 3x slower and Structured Skip-gram 2x slower than their baselines—they remain highly scalable. Future work may address parameter sparsity in smaller datasets and explore capturing long-distance dependencies beyond a fixed window.
Summary updated