Neural word embeddings with multiplicative feature interactions for tensor-based compositions

Joo-Kyung Kim, Marie-Catherine de Marneffe, Eric Fosler-Lussier

Abstract

Standard word embeddings like CBOW are typically trained by averaging context words. However, many downstream tasks combine words using multiplicative or complex tensor operations. This paper introduces extensions to the CBOW model that incorporate multiplicative and tensor-based interactions during training. We demonstrate that matching the training mathematics to the evaluation mathematics significantly improves phrase-level semantic representations, particularly in tasks requiring nuanced word interactions.

Introduction

In distributional semantics, words are represented as vectors based on their context. The popular Continuous Bag-of-Words (CBOW) model predicts a target word by taking the mean (an additive operation) of its surrounding context vectors. However, representing the meaning of a whole phrase (e.g., "cats like milk") often requires more sophisticated "compositional" methods like pointwise multiplication or tensor products. These methods capture how words interact, such as how the meaning of "bank" changes when paired with "river" versus "money."

A fundamental mismatch exists in current practices: word vectors are often trained assuming addition but evaluated using multiplication. This paper explores whether training word embeddings with multiplicative interactions from the start improves performance on phrase-level tasks.

Methods

The authors propose several extensions to the standard CBOW architecture to bridge the gap between training and evaluation. Instead of just using the mean of context word projections ($p_i$), they experiment with several new aggregation types:

  • Pointwise Multiplication (mult): Multiplying all context vectors element-wise.
  • Hybrid Models: Combining the mean and the product via addition or concatenation.
  • Neighbor Projections: Projecting the concatenation of immediate neighbors ($p_{i-1}$ and $p_{i+1}$).
  • Tensor Products (nbr_outer_prj): Taking the outer product of immediate neighbors to capture all feature-to-feature interactions, then projecting back to the standard vector size.
Proposed training architectures and their mathematical expressions.

To maintain training stability against vanishing or exploding gradients inherent in multiplication, the authors employed gradient clipping and hierarchical softmax as the objective function.

Key Results

The models were trained on the British National Corpus (BNC) and evaluated on benchmarks including Transitive Verb Similarity, Sentence Similarity, Paraphrase Detection (MSRP), and Dialog Act Tagging. The results indicate that alignment is essential: the best-performing model is typically the one where the training operation matches the evaluation operation.

Performance comparison on similarity tasks across different training and evaluation methods.

As seen in the results table, for sentence similarity using multiplicative composition, the mult training model achieved a Spearman’s $\rho$ of 0.625, compared to a dismal 0.062 for the standard CBOW baseline. Similarly, on the MSRP dataset, the mult model yielded an accuracy of 0.652, vastly outperforming the baseline's 0.393. The mean + nbr_outer_prj model showed consistent performance across tensor-based tasks, often outperforming the baseline by 5-10 points in Spearman correlation.

Conclusion

Training word embeddings with multiplicative interactions allows the model to learn vectors that are mathematically robust for these operations. Vectors trained with the mult model exhibited a higher mean and fewer negative elements, leading to more stable results during element-wise multiplication. While tensor products increase computational complexity and are currently limited to fixed windows, they offer a significant path forward for capturing nuanced semantic interactions in phrase-level tasks. Future work may involve using dependency parses to identify subject-object pairs more accurately across larger context windows.

Summary updated

Abstract

Categorical compositional distributional models unify compositional formal semantic models and distributional models by composing phrases with tensor-based methods from vector representations. For the tensor-based compositions, Milajevs et al. (2014) showed that word vectors obtained from the continuous bag-of-words (CBOW) model are competitive with those from co-occurrence based models. However, because word vectors from the CBOW model are trained assuming additive interactions between context words, the word composition used for the training mismatches to the tensor-based methods used for evaluating the actual compositions including pointwise multiplication and tensor product of context vectors. In this work, we show whether the word embeddings from extended CBOW models using multiplication or tensor product between context words, reflecting the actual composition methods, can show better performance than those from the baseline CBOW model in actual tasks of compositions with multiplication or tensor-based methods.
BibTeX
@inproceedings{kim2015neural,
  author = {Joo-Kyung Kim and Marie-Catherine de Marneffe and Eric Fosler-Lussier},
  title = {{Neural word embeddings with multiplicative feature interactions for tensor-based compositions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/w15-1520},
}
Figure