Second-Order Word Embeddings from Nearest Neighbor Topological Features

Denis Newman-Griffis, Eric Fosler-LussierarXiv 2017 · OpenReview

Abstract

We introduce second-order vector representations of words, induced from nearest neighborhood topological features in pre-trained contextual word embeddings. We then analyze the effects of using second-order embeddings as input features in two deep natural language processing models, for named entity recognition and recognizing textual entailment, as well as a linear model for paraphrase recognition. Surprisingly, we find that nearest neighbor information alone is sufficient to capture most of the performance benefits derived from using pre-trained word embeddings. Furthermore, second-order embeddings are able to handle highly heterogeneous data better than first-order representations, though at the cost of some specificity. Additionally, augmenting contextual embeddings with second-order information further improves model performance in some cases. Due to variance in the random initializations of word embeddings, utilizing nearest neighbor features from multiple first-order embedding samples can also contribute to downstream performance gains. Finally, we identify intriguing characteristics of second-order embedding spaces for further research, including much higher density and different semantic interpretations of cosine similarity.

Introduction

Standard word embeddings such as word2vec or GloVe are fundamental to modern Natural Language Processing (NLP). However, these embeddings are typically initialized randomly, meaning that while the relative relationships between words remain consistent across training runs, the absolute numerical coordinates change every time. This suggests that the semantic "meaning" of a word in vector space may reside more in its local topology—its neighborhood structure—than in its specific coordinates. This paper investigates whether a representation built exclusively from these neighborhood relationships can capture the semantic richness of the original embeddings.

Methods

The researchers propose a two-stage pipeline to generate second-order embeddings:

  1. Graph Induction: Using a set of pre-trained "first-order" embeddings, a $k$-nearest neighbor ($k$-NN) graph is constructed. For each word in the vocabulary, directed edges are drawn to its $k$ closest neighbors based on cosine similarity.
  2. Graph Embedding: This graph is treated as a social network where nodes are words. The node2vec algorithm is then applied to learn new vector representations based on random walks through this neighborhood graph.
The process of inducing a 3-NN graph from first-order embedding coordinates.

To improve robustness against the noise of random initialization, the authors also explore multi-sample graphs, which aggregate neighbors from multiple independent word2vec training runs into a single weighted graph to "smooth out" the noise inherent in random initialization.

Key Insights and Results

The study reveals several critical properties of second-order embeddings regarding performance and space density.

Performance and Robustness

Second-order embeddings perform remarkably well despite having zero knowledge of the original text. In Named Entity Recognition (NER) tasks on the CoNLL-2003 dataset, second-order embeddings achieved an F1 score of 86.66, nearly matching the word2vec baseline of 87.56. Notably, they excelled in the "Miscellaneous" (MISC) category, where multi-sample second-order embeddings improved precision by 8.4% over the baseline, suggesting superior handling of heterogeneous data.

Embedding Space Density

A striking difference between the two spaces is their density. First-order embeddings often result in "isolated" words with low similarity to their neighbors. In contrast, the second-order space is significantly denser.

Comparison of cosine similarity distributions between first-order and second-order embeddings, showing a significant increase in density in the second-order space.

As shown in the similarity distributions, the lowest similarity between neighbors typically jumps from approximately 0.24 in the first-order space to over 0.75 in the second-order space. This indicates a shift from capturing strict synonyms to capturing broader "relatedness." For instance, while a first-order model might group "zucchini" with "squash," a second-order model might also pull in "garlic" or "thyme"—words related by a broader culinary context.

Conclusion

This research demonstrates that the local topology of word embedding spaces carries the majority of semantic information. Second-order embeddings provide a robust, dense alternative that can enhance performance in complex tasks like NER and Textual Entailment, particularly when dealing with diverse data. While there is a trade-off between broad relatedness and fine-grained specificity, the neighborhood-based approach offers a promising path for de-noising and aligning embedding spaces.

Summary updated

Abstract

We introduce second-order vector representations of words, induced from nearest neighborhood topological features in pre-trained contextual word embeddings. We then analyze the effects of using second-order embeddings as input features in two deep natural language processing models, for named entity recognition and recognizing textual entailment, as well as a linear model for paraphrase recognition. Surprisingly, we find that nearest neighbor information alone is sufficient to capture most of the performance benefits derived from using pre-trained word embeddings. Furthermore, second-order embeddings are able to handle highly heterogeneous data better than first-order representations, though at the cost of some specificity. Additionally, augmenting contextual embeddings with second-order information further improves model performance in some cases. Due to variance in the random initializations of word embeddings, utilizing nearest neighbor features from multiple first-order embedding samples can also contribute to downstream performance gains. Finally, we identify intriguing characteristics of second-order embedding spaces for further research, including much higher density and different semantic interpretations of cosine similarity.
BibTeX
@misc{newmangriffis2017second,
  author = {Denis Newman-Griffis and Eric Fosler-Lussier},
  title = {{Second-Order Word Embeddings from Nearest Neighbor Topological Features}},
  year = {2017},
  eprint = {1705.08488},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1705.08488},
}
Figure