On Learning Word Embeddings From Linguistically Augmented Text Corpora

Amila Silva, Chathurika AmarathungaIwcs 2019 · DOI

TL;DR

This paper introduces a novel method for generating effective word embeddings from limited text data by linguistically augmenting sentences. By constructing a network of words within a sentence that captures both linear and dependency relationships, and then performing biased random walks, the authors create more diverse training data for word embedding models, leading to improved performance on downstream NLP tasks, especially in low-resource scenarios.

What Problem Does This Paper Address?

Word embeddings, which represent words as dense vectors, are fundamental to modern Natural Language Processing (NLP). They capture semantic and syntactic relationships, boosting performance across various tasks. However, traditional methods like word2vec and GloVe struggle to produce high-quality embeddings when the training corpus is small or sparse. This is particularly problematic for domain-specific applications where large datasets are often unavailable. Existing approaches to address data scarcity, such as domain adaptation, still require substantial amounts of data from a source domain. Furthermore, most current methods capture only one type of linguistic context (e.g., linear proximity), missing out on richer relationships like syntactic dependencies.

Methods

The core of this paper's contribution lies in its method for generating more diverse training data for word embedding models. Instead of relying solely on the original text, the authors propose to linguistically augment sentences.

  1. Linguistic Network Construction: For each sentence, a network is built where words are nodes. Edges represent two types of relationships:
  2. Linear links: Connect adjacent words in the sentence.
  3. Dependency links: Connect words based on their grammatical dependencies. This network captures richer contextual information than simple linear proximity.

  4. Biased Random Walks: The authors then perform random walks on this linguistic network. These walks are "biased" to prioritize different edge types (linear vs. dependency) using hyperparameters p and q. By adjusting these parameters, the walks can be steered to explore either predominantly linear contexts or dependency-based contexts, or a combination of both. The goal is to generate multiple "versions" of each sentence, each representing a different linguistic perspective.

  5. Word Embedding Training: The sequences of words generated by these random walks from all sentences are then treated as a single, augmented corpus. This augmented corpus is fed into a standard word2vec skip-gram model to learn word embeddings. The intuition is that by providing more varied "contexts" through the random walks, the word2vec model can learn more robust and informative representations, especially from limited data.

Datasets

The paper uses two main types of datasets for evaluation:

  • Word Similarity/Relatedness Task:
    • English Wikipedia: A subset of approximately 1.9 million sentences was used.
  • MEN dataset: Used for hyperparameter tuning and as one of the evaluation benchmarks.
  • SIMLEX-999 and WS-353 datasets: Used for evaluating word similarity.
  • Malware-Domain Specific Sentence Classification:
    • APTnotes corpus: A collection of 193 reports (2010-2015) related to malware campaigns, activity, and software. This corpus was extracted from PDF files and processed to yield approximately 27,500 sentences.
  • Wikipedia corpus: Used as a source domain for cross-domain experiments.

Key Insights and Ideas

  • Linguistic structure is key for low-resource NLP: The paper strongly suggests that leveraging linguistic information (like dependency parsing) is crucial for learning effective word embeddings when data is scarce.
  • Augmenting data with linguistic context pays off: Generating multiple "views" of a sentence through random walks on a linguistic network significantly improves embedding quality compared to using the raw, limited corpus.
  • Balancing context types is optimal: The results indicate that a balanced approach, capturing both linear and dependency contexts (around q=1), yields the best performance, suggesting that words benefit from understanding their local neighbors as well as their grammatical roles.
  • Random walks are a powerful data generation tool: The technique of using biased random walks on graph structures is shown to be an effective way to create synthetic training data for NLP tasks.

Key Results

  • Word Similarity Task: The proposed model consistently outperforms baseline methods (SVD, word2vec, GloVe, and dependency-based embeddings like Ext) across three standard datasets (SIMLEX-999, WS-353, MEN). For instance, on the MEN dataset, the proposed model achieved a Spearman's rank correlation of 0.6293, outperforming word2vec (0.5078) and GloVe (0.5799).
  • Malware-Domain Specific Sentence Classification: The model achieved state-of-the-art results on this task, outperforming baselines with both static and fine-tuned embeddings. The CNN-static version achieved 83.12% accuracy, and the CNN-non-static version reached 83.55%, demonstrating the quality of the generated embeddings for domain-specific tasks.
  • Parameter Sensitivity: The paper highlights that the model's improvement over word2vec is most pronounced when the corpus size is small. The optimal performance is achieved when both linear and dependency contexts are captured (q=1), and increasing the number of random walks per sentence improves performance up to a point (around 3 walks).

Limitations and Future Work

  • Not all contexts are equally useful: The authors acknowledge that different types of linguistic contexts (dependency, linear) might not be equally beneficial for all words.
  • Sophistication of walking: A more sophisticated random walk approach on linguistic structures could be explored.
  • Incorporating other linguistic structures: The paper suggests future work could involve integrating other linguistic features like co-reference resolution, Named Entity Recognition (NER), and Part-of-Speech (POS) tag sequences into the proposed model.

Summary updated

BibTeX
@inproceedings{silva2019learning,
  author = {Amila Silva and Chathurika Amarathunga},
  title = {{On Learning Word Embeddings From Linguistically Augmented Text Corpora}},
  booktitle = {Iwcs},
  year = {2019},
  doi = {10.18653/v1/w19-0508},
}
Figure