Incorporating Subword Information into Matrix Factorization Word Embeddings

Alexandre Salle, Aline VillavicencioNAACL 2018

TL;DR

This paper explores how to inject subword information (like character n-grams or morphemes) into matrix factorization-based word embedding models, similar to how fastText does for predictive models. They found that while subword information improves the representation of rare words and syntactic understanding, it doesn't significantly boost performance on downstream tasks when using standard evaluation setups.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) heavily relies on word embeddings, which represent words as dense vectors. Traditional methods often struggle with two main issues: 1) they treat each word as an independent entity, making it hard to share learned information across similar words, and 2) they cannot represent out-of-vocabulary (OOV) words. While predictive models like word2vec have been enhanced with subword information (e.g., fastText) to address these problems, the impact of subword information on counting-based word embedding models, like LexVec, has been less explored. This paper aims to bridge that gap by investigating how incorporating subword information into matrix factorization models affects both in-vocabulary (IV) and OOV word representations.

Methods

The authors adapt the LexVec model, a popular matrix factorization approach, to incorporate subword information. Their core idea is to represent a word's vector as the sum of its own unique vector and the vectors of its constituent subwords. They experiment with two types of subwords:

  • Character n-grams: Similar to fastText, they break words down into overlapping sequences of characters (e.g., "cat" might yield "ca", "at", "cat").
  • Unsupervised Morphemes: They use Morfessor, an unsupervised tool, to segment words into meaningful linguistic units (e.g., "subsequent" might be segmented into "sub" and "sequent").

They then compare these subword-enhanced LexVec models (LV-N for n-grams, LV-M for morphemes) against the original word-level LexVec (LV), as well as the predictive models Skip-gram (SG) and fastText (FT).

Datasets

  • Training Data: A 2015 dump of Wikipedia, lowercased and filtered to include only alphanumeric characters. Words appearing at least 100 times were kept, resulting in a vocabulary of 303,517 words.
  • Evaluation Datasets:
    • Intrinsic Evaluation:
      • Word Similarity: WordSim-353 Similarity (WS-Sim), Relatedness (WS-Rel), SimLex-999 (SimLex), and Rare Word (RW) datasets.
  • Word Analogy: Google semantic (GSem) and syntactic (GSyn) analogies, and the Microsoft syntactic analogies (MSR) dataset.
  • Downstream Tasks (VecEval suite): Chunking, sentiment classification, and natural language identification (NLI). The authors specifically used tasks with freely available data and fixed settings where embeddings were not tunable.
  • OOV Word Evaluation: Qualitative nearest-neighbor analysis for generated OOV word representations.

Key Insights and Ideas

  • Subwords Help Rare Words: Just like in predictive models, incorporating subword information significantly improves the representation of rare words. This is crucial because rare words are often important for capturing nuanced meaning.
  • Syntax vs. Semantics Trade-off: While both n-grams and morphemes improve syntactic understanding (evidenced by gains in GSyn and MSR tasks), unsupervised morphemes seem to preserve semantics better than simple n-grams. This suggests that linguistically motivated subword units might offer a more balanced representation.
  • Downstream Task Disconnect: A surprising finding is that despite improvements in intrinsic evaluations and OOV handling, the subword-enhanced models did not show significant improvements on downstream tasks. The authors attribute this to the fact that most downstream tasks primarily use frequent words, and OOV words are often mapped to a generic "(unk)" token, thus not fully leveraging the subword capabilities.
  • OOV Coherence: Both n-gram and morpheme-based models can generate coherent nearest neighbors for OOV words, demonstrating their ability to construct representations from subword components. However, n-grams proved more robust than morphemes in this regard.

Key Results

  • Rare Word Improvement: The RW dataset showed clear gains for subword models, indicating better representation of infrequent words.
  • Syntactic Gains: Subword models (LV-N and LV-M) showed significant improvements on GSyn and MSR tasks, suggesting they capture syntactic function better (e.g., suffixes like "-ly").
  • Downstream Task Stagnation: On most downstream tasks (chunking, sentiment, NLI), subword models showed no statistically significant improvement over word-level models. The only exception was a very small gain in chunking.
  • OOV Nearest Neighbors: For OOV words like "hellooo", "marvelicious", and "rereread", both LV-N and FT found coherent neighbors. LV-M also performed well but was sometimes sensitive to segmentation errors. For "tuzread", all models struggled, suggesting a need to down-weight very rare subwords.

Limitations and Future Work

  • Downstream Task Limitations: The authors acknowledge that the current evaluation setup for downstream tasks doesn't fully utilize the potential of subword models, particularly for OOV words.
  • OOV Handling: While subword models generate representations for OOV words, the current setup maps them to a generic token in downstream tasks. Future work should investigate how to integrate these OOV representations to improve overall task performance.
  • Tunable Representations: The paper suggests that by identifying semantic vs. syntactic subwords and weighting them accordingly, it might be possible to create tunable representations. This could be achieved without supervision by leveraging corpus statistics.
  • Trade-off Exploration: Further research is needed to explore the trade-off between capturing semantics and syntax when using subword information.

Summary updated

BibTeX
@inproceedings{salle2018incorporating,
  author = {Alexandre Salle and Aline Villavicencio},
  title = {{Incorporating Subword Information into Matrix Factorization Word Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure