Can Network Embedding of Distributional Thesaurus be Combined with Word Vectors for Better Representation?

Abhik Jana, Pawan Goyal

Abstract

Distributed representations of words learned from text have proved to be successful in various natural language processing tasks in recent times. While some methods represent words as vectors computed from text using predictive models (Word2vec) or dense count-based models (GloVe), others attempt to represent these in a distributional thesaurus network structure where the neighborhood of a word is a set of words having adequate context overlap. Being motivated by recent surge of research in network embedding techniques, we turn a distributional thesaurus network into dense word vectors and investigate the usefulness of distributional thesaurus embedding in improving overall word representation. This is the first attempt where we show that combining the proposed word representation obtained by distributional thesaurus embedding with the state-of-the-art word representations helps in improving the performance by a significant margin when evaluated against NLP tasks like word similarity and relatedness, synonym detection, and analogy detection. Additionally, we show that even without using any handcrafted lexical resources, we can achieve representations with comparable performance to those using lexical resources.

Introduction

In Natural Language Processing (NLP), word representation is a fundamental challenge. Current approaches generally fall into two categories: predictive/count-based vectors (like Word2vec and GloVe) which capture local context, and network-based structures like Distributional Thesauri (DT) which capture structural relationships. While standard vectors are powerful, they often miss the nuanced hierarchical information found in thesauri. Conversely, human-curated resources like WordNet are high-quality but expensive and difficult to scale. This research investigates whether automatically generated DTs can be transformed into vectors and fused with standard word vectors to create a superior representation without manual annotation.

Methodology

The proposed approach utilizes a three-stage pipeline to enhance word representations:

  1. DT Network Construction: Using the Google Books corpus, a network is constructed where words are nodes. Edges are established based on context overlap (bigram distributions), creating a graph-based similarity structure.
A sample Distributional Thesaurus network showing semantic relationships and context overlap weights between words.
  1. Network Embedding: Graph embedding algorithms—specifically DeepWalk (D2V-D), LINE (D2V-L), and node2vec (D2V-N)—are applied to the DT to generate 128-dimensional dense vectors that capture the global topology of the thesaurus.
  2. Vector Fusion: The DT-based vectors are combined with pre-trained 300-dimensional GloVe or Word2vec vectors. Two strategies were tested: simple Concatenation (CC) and Principal Component Analysis (PCA). PCA was found to be more effective, likely because it filters noise and identifies the most informative features from both semantic sources.

Results and Analysis

The models were evaluated on standard benchmarks for word similarity, synonym detection, and analogy detection.

Performance Gains

The combined representations consistently outperformed standalone models across nearly all datasets. As shown in the table below, the fusion of GloVe with network embeddings via PCA leads to significant improvements in similarity scores.

Performance comparison of GloVe combined with DT embeddings (DeepWalk and node2vec) using Concatenation (CC) and PCA.

For example, on the MC-30 dataset, standalone GloVe achieved a score of 0.799, while the combined model reached 0.89. Similar trends were observed in synonym detection (TOEFL-N), where accuracy rose from 82.6% (GloVe) to 86.9% (combined).

Key Insights

  • DT as a WordNet Proxy: Embeddings from automatically generated DTs perform comparably to those derived from WordNet, suggesting that the "human bottleneck" of manual lexicography can be bypassed.
  • Structural Complementarity: Text-based vectors capture usage, while DT embeddings capture hierarchical position. Their combination provides a more holistic semantic view.
  • Fusion vs. Retrofitting: The PCA-based fusion method proved more effective than standard "retrofitting" techniques, indicating that treating the DT as a primary embedding source is superior to using it as a post-processing constraint.

Conclusion

This study demonstrates that network embeddings of distributional thesauri provide valuable structural information that is complementary to standard text-based word vectors. By fusing these two sources using PCA, we achieve state-of-the-art performance on various intrinsic NLP tasks without the need for handcrafted resources. Future work will extend these experiments to other parts of speech beyond nouns and evaluate performance on extrinsic downstream tasks like sentiment analysis and text classification.

Summary updated

Abstract

Distributed representations of words learned from text have proved to be successful in various natural language processing tasks in recent times. While some methods represent words as vectors computed from text using predictive model (Word2vec) or dense count based model (GloVe), others attempt to represent these in a distributional thesaurus network structure where the neighborhood of a word is a set of words having adequate context overlap. Being motivated by recent surge of research in network embedding techniques (DeepWalk, LINE, node2vec etc.), we turn a distributional thesaurus network into dense word vectors and investigate the usefulness of distributional thesaurus embedding in improving overall word representation. This is the first attempt where we show that combining the proposed word representation obtained by distributional thesaurus embedding with the state-of-the-art word representations helps in improving the performance by a significant margin when evaluated against NLP tasks like word similarity and relatedness, synonym detection, analogy detection. Additionally, we show that even without using any handcrafted lexical resources we can come up with representations having comparable performance in the word similarity and relatedness tasks compared to the representations where a lexical resource has been used.
BibTeX
@inproceedings{jana2018can,
  author = {Abhik Jana and Pawan Goyal},
  title = {{Can Network Embedding of Distributional Thesaurus be Combined with Word Vectors for Better Representation?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-1043},
}
Figure