Abstract
Standard word embeddings like CBOW are typically trained by averaging context words. However, many downstream tasks combine words using multiplicative or complex tensor operations. This paper introduces extensions to the CBOW model that incorporate multiplicative and tensor-based interactions during training. We demonstrate that matching the training mathematics to the evaluation mathematics significantly improves phrase-level semantic representations, particularly in tasks requiring nuanced word interactions.
Introduction
In distributional semantics, words are represented as vectors based on their context. The popular Continuous Bag-of-Words (CBOW) model predicts a target word by taking the mean (an additive operation) of its surrounding context vectors. However, representing the meaning of a whole phrase (e.g., "cats like milk") often requires more sophisticated "compositional" methods like pointwise multiplication or tensor products. These methods capture how words interact, such as how the meaning of "bank" changes when paired with "river" versus "money."
A fundamental mismatch exists in current practices: word vectors are often trained assuming addition but evaluated using multiplication. This paper explores whether training word embeddings with multiplicative interactions from the start improves performance on phrase-level tasks.
Methods
The authors propose several extensions to the standard CBOW architecture to bridge the gap between training and evaluation. Instead of just using the mean of context word projections ($p_i$), they experiment with several new aggregation types:
- Pointwise Multiplication (
mult): Multiplying all context vectors element-wise. - Hybrid Models: Combining the mean and the product via addition or concatenation.
- Neighbor Projections: Projecting the concatenation of immediate neighbors ($p_{i-1}$ and $p_{i+1}$).
- Tensor Products (
nbr_outer_prj): Taking the outer product of immediate neighbors to capture all feature-to-feature interactions, then projecting back to the standard vector size.
To maintain training stability against vanishing or exploding gradients inherent in multiplication, the authors employed gradient clipping and hierarchical softmax as the objective function.
Key Results
The models were trained on the British National Corpus (BNC) and evaluated on benchmarks including Transitive Verb Similarity, Sentence Similarity, Paraphrase Detection (MSRP), and Dialog Act Tagging. The results indicate that alignment is essential: the best-performing model is typically the one where the training operation matches the evaluation operation.
As seen in the results table, for sentence similarity using multiplicative composition, the mult training model achieved a Spearman’s $\rho$ of 0.625, compared to a dismal 0.062 for the standard CBOW baseline. Similarly, on the MSRP dataset, the mult model yielded an accuracy of 0.652, vastly outperforming the baseline's 0.393. The mean + nbr_outer_prj model showed consistent performance across tensor-based tasks, often outperforming the baseline by 5-10 points in Spearman correlation.
Conclusion
Training word embeddings with multiplicative interactions allows the model to learn vectors that are mathematically robust for these operations. Vectors trained with the mult model exhibited a higher mean and fewer negative elements, leading to more stable results during element-wise multiplication. While tensor products increase computational complexity and are currently limited to fixed windows, they offer a significant path forward for capturing nuanced semantic interactions in phrase-level tasks. Future work may involve using dependency parses to identify subject-object pairs more accurately across larger context windows.
Summary updated