Getting in Shape: Word Embedding SubSpaces

Tianyuan Zhou, João Sedoc, Jordan RoduIJCAI 2019 · DOI

TL;DR

This paper investigates why aligning word embeddings (mapping them into a common space) works and when it fails, particularly focusing on the "shape" of the embedding space. They propose a theoretical framework showing that the target embedding's singular value structure dictates the aligned embedding's properties, and empirically demonstrate that target embedding isotropy is crucial for successful alignment, even suggesting that adding isotropic noise can improve results.

What Problem Does This Paper Address?

Word embeddings, like Word2Vec or GloVe, represent words as dense vectors in a high-dimensional space. These embeddings capture semantic relationships, but often need to be aligned to be used across different tasks or languages (e.g., domain adaptation, machine translation). This alignment process, typically done with linear transformations, is known to work reasonably well, but the underlying reasons for its success and limitations are not fully understood. Specifically, the paper questions how the geometric "shape" of the source and target embedding spaces influences the alignment outcome. They hypothesize that the target embedding's intrinsic properties, like its isotropy (uniformity in all directions), play a critical role, a connection that has been underexplored.

Methods

The authors propose a theoretical framework for understanding linear word embedding alignment. They derive results showing that when aligning a source embedding $X$ to a target embedding $Y$ using a linear transformation $W$ (i.e., $Y \approx XW$), the singular value structure of the source embedding is largely discarded. Instead, the singular value structure of the aligned embedding is primarily determined by the singular value structure of the target embedding.

Key theoretical contributions include:

  • Lemma 1: Relates the singular values of the aligned embedding to the singular values of the source and target embeddings.
  • Theorem 1: Guarantees the existence of a lower-bounded singular value in the aligned embedding, influenced by the target's singular values and the overlap of singular vector spaces.
  • Proposition 1: Shows that stretching a vector cloud unequally in different directions (analogous to an anisotropic target embedding) significantly reduces the correlation of pairwise distances between vectors. This implies that anisotropic target embeddings can distort semantic relationships after alignment.

Empirically, they evaluate alignment performance on semantic similarity tasks by systematically comparing various alignment strategies, including using conceptor negation (a technique to make embeddings more isotropic) on source, target, or aligned embeddings.

Datasets

The paper uses standard distributional word representations for experiments:

  • Source Embeddings: word2vec (Google News), GloVe (Common Crawl), FastText (Common Crawl without subword).
  • Target Representations:
    • Other distributional word representations (GloVe, FastText, word2vec).
  • Non-distributional word vectors (LSA, Eigenwords, dependency word embeddings like CBOW and Skip-Gram).
  • Permuted versions of these representations.
  • Designed noise targets (isotropic Gaussian noise, rank-1 noise).

Evaluation is performed on semantic similarity tasks (e.g., EN-MEN-TR-3k, EN-MTurk-287, EN-SIMLEX-999, EN-WS-353-ALL, EN-RG-65, EN-SimVerb-3500, EN-RW-STANFORD) and concept categorization tasks.

Key Insights and Ideas

  • Target Dominance: The alignment process, particularly linear alignment, effectively "forgets" the singular value structure of the source embedding. The target embedding's singular values and their corresponding directions are what primarily shape the aligned embedding. This is a crucial insight for understanding why alignment works.
  • Isotropy is Key: The "shape" of the target embedding space matters immensely. Isotropic target embeddings (where vectors are roughly equally distributed in all directions) lead to better alignment performance on semantic tasks. Anisotropic (non-uniform) target embeddings can distort semantic relationships.
  • Conceptor Negation as a Fix: Applying conceptor negation to target embeddings can make them more isotropic, leading to improved alignment results. This suggests a practical way to enhance alignment by pre-processing the target.
  • Non-distributional vs. Distributional: Aligning distributional embeddings to non-distributional ones is challenging. The paper suggests that concatenation might be a better strategy than alignment for combining these types of representations due to their fundamentally different geometric structures.
  • Noise as a Control: Aligning with isotropic Gaussian noise minimally degrades performance, while aligning with highly anisotropic rank-1 noise destroys semantic information, reinforcing the importance of target isotropy.

Key Results

  • Aligning distributional word embeddings to other distributional embeddings generally yields high $R^2$ values, indicating good alignment.
  • Aligning distributional embeddings to non-distributional embeddings results in lower $R^2$ values, especially when the target is not conceptor-negated. Figure 2 shows that singular vectors corresponding to larger singular values of non-distributional targets contribute more to the alignment.
  • Conceptor negation of the target embedding consistently improves semantic similarity task scores compared to aligning with the original target.
  • Aligning with permuted targets often leads to the worst results, as it breaks the semantic correspondence.
  • Aligning with a permuted and conceptor-negated target can yield surprisingly good semantic similarity scores, suggesting that tempering the target's singular value information makes the aligned vectors more isotropic and thus better for semantic tasks.

Limitations and Future Work

  • Focus on Linear Alignment: The theoretical framework and much of the empirical work focus on linear alignment. While common, other alignment methods exist.
  • Theoretical Guarantees: While theorems provide bounds and guarantees, the practical implications of these bounds for specific alignment scenarios could be further explored.
  • Task-Specific Alignment: The paper primarily evaluates on semantic similarity tasks. Exploring how these findings generalize to other NLP tasks (e.g., translation, classification) would be valuable.
  • Understanding "Why": The authors state that their work lays the groundwork for understanding alignment and how word representations encode information. Future work could delve deeper into how specific geometric properties of embeddings relate to specific linguistic phenomena.
  • Concatenation vs. Alignment: The paper suggests concatenation might be better for combining distributional and non-distributional embeddings. Further research could explore optimal strategies for such combinations.

Summary updated

BibTeX
@inproceedings{zhou2019getting,
  author = {Tianyuan Zhou and João Sedoc and Jordan Rodu},
  title = {{Getting in Shape: Word Embedding SubSpaces}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/761},
}
Figure