Learning Embeddings for Rare Words Leveraging Internet Search Engine and Spatial Location Relationships

Xiaotao Li, Shujuan You, Yawen Niu, Wai ChenACL 2021

TL;DR

This paper introduces a novel two-step algorithm to learn high-quality word embeddings for rare words, which are often poorly represented by traditional methods. By leveraging Internet search engines to find related words and then fine-tuning their positions in a vector space based on relevance, the authors achieve state-of-the-art performance on word relatedness tasks.

What Problem Does This Paper Address?

Word embeddings, like Word2Vec and GloVe, have revolutionized Natural Language Processing (NLP) by representing words as dense vectors. However, these methods heavily rely on word frequency in the training corpus. This means that rare words (low-frequency or unseen words) are often poorly represented or entirely missing, limiting their utility in downstream NLP tasks. This is a significant problem because natural language distributions, like Zipf's law, mean that most words in a corpus are rare. Existing approaches to tackle this issue include using subword information (like character n-grams) or relying on external lexical resources (like WordNet). While these methods offer improvements, they have limitations: subword methods might not capture full word semantics, and lexical resources can have incomplete coverage. This paper aims to address the challenge of learning accurate and comprehensive embeddings for rare words by combining the vastness of the Internet with a novel fine-tuning strategy.

Methods

The proposed algorithm consists of two main stages: coarse-tuning and fine-tuning.

Coarse-Tuning: 1. Internet Search & Related Word Extraction: For a given rare word, the algorithm uses an Internet search engine to find relevant webpages. It then extracts the titles of the top-m webpages and parses them to identify a set of n semantically related words. To mitigate noise, common irrelevant words (like "definition," "synonyms," etc.) are filtered out. 2. Initial Embedding: The initial embedding for the rare word is computed by averaging the pre-trained embeddings of these n related words. This provides a coarse-grained representation.

Fine-Tuning: 1. Spatial Location Constraints: The core idea here is that words with higher semantic relatedness to the rare word should be closer to it in the vector space. The algorithm leverages the ranked list of related words from the coarse-tuning stage. 2. Iterative Adjustment: The coarse-grained embedding is iteratively adjusted. If a word wi is ranked higher than wj (meaning wi is more related to the rare word w), the algorithm ensures that the distance between w and wi is smaller than the distance between w and wj. This is achieved by moving the rare word's vector in the direction of the difference between the vectors of more relevant words and less relevant words, effectively pulling it closer to its true semantic location.

Datasets

  • Training Corpus: English Wikipedia dump (April 1, 2015).
  • Benchmark Datasets (English):
    • Stanford Rare Word (RW) dataset (general domain)
    • Cambridge Rare Word (Card-660) dataset (general domain)
    • UMNSRS dataset (biomedical field)
    • MayoSRS dataset (biomedical field)
  • Benchmark Datasets (Chinese):
    • wordsim-240
    • wordsim-296

Key Insights and Ideas

  • Internet as a Vast Lexicon: The Internet, with its billions of pages, serves as a more comprehensive source for finding semantically related words for rare terms than traditional, limited lexical resources.
  • Leveraging Relevance Ranking: The key innovation lies in using the order of relevance of extracted words, not just their presence, to fine-tune the rare word's embedding. This spatial relationship constraint is crucial for accuracy.
  • Coarse-to-Fine Strategy: The two-step approach (initial averaging followed by iterative refinement) effectively balances broad coverage with precise semantic placement.
  • Robustness to Lexicon Coverage: Unlike methods that strictly depend on lexical resources, this approach is less susceptible to gaps in those resources because it primarily relies on Internet search results.

Key Results

  • State-of-the-Art Performance: The proposed algorithm achieved state-of-the-art performance on the word relatedness task across four English benchmark datasets, outperforming established methods like FastText, FCM, SemLand, Align, ELMO, and BERT.
  • Improved Quality: The fine-tuning process significantly enhanced the quality of embeddings compared to the coarse-grained embeddings alone.
  • High Vocabulary Coverage: The algorithm demonstrated excellent coverage for rare words, comparable to or better than pre-trained language models like ELMO and BERT, and significantly better than subword-based methods on domain-specific datasets.
  • Scalability to Chinese: The approach showed good performance on Chinese rare words, indicating its potential for multilingual applications.

Limitations and Future Work

  • Hyper-parameter Sensitivity: The performance is influenced by hyper-parameters like the number of retrieved records (m) and related words (n). The paper identifies optimal ranges (e.g., m=100, n=5) but acknowledges that these might vary.
  • Future Work: The authors plan to extend their work to learn contextualized rare word representations, potentially by combining their approach with attentive mimicking (AM) architectures. They also aim to explore different search engines and languages to further improve multilingual capabilities.

Summary updated

BibTeX
@inproceedings{li2021learning,
  author = {Xiaotao Li and Shujuan You and Yawen Niu and Wai Chen},
  title = {{Learning Embeddings for Rare Words Leveraging Internet Search Engine and Spatial Location Relationships}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure