Unsupervised Post-Processing of Word Vectors via Conceptor Negation.

Tianlin Liu, Lyle Ungar, João Sedoc

TL;DR

This paper introduces "Conceptor Negation" (CN), a novel unsupervised method to refine word vectors by suppressing features that correlate with word frequency. CN leverages matrix conceptors to achieve a "softer" noise reduction than prior methods, leading to significant improvements on various intrinsic lexical tasks and a downstream dialogue state tracking task.

What Problem Does This Paper Address?

Word vectors, like Word2Vec and GloVe, are fundamental to modern Natural Language Processing (NLP). However, these vectors often encode information that is not directly related to semantic meaning, such as word frequency. This "noise" can hinder the performance of downstream NLP tasks. While previous methods like "all-but-the-top" (ABTT) attempted to address this by removing principal components (PCs) associated with high variance (and thus, likely word frequency), they did so in a "hard" manner, either keeping or discarding features entirely. This can lead to either insufficient noise removal or excessive information loss. This paper aims to provide a more nuanced and effective unsupervised approach to post-process word vectors.

Methods

The core of this paper is the Conceptor Negation (CN) algorithm. It builds upon the concept of matrix conceptors, which are a family of regularized identity maps that can represent linear transformations. CN uses a negated conceptor (-C = I - C) to "gate away" variances explained by the leading principal components of word vectors.

Here's a breakdown of the CN approach:

  1. Estimate a Conceptor (C): Based on the input word vectors, a conceptor C is estimated. This conceptor captures the linear subspace where the word vectors' variances are highest.
  2. Negate the Conceptor: The negated conceptor -C is computed. Intuitively, this projects data onto the orthogonal complement of the subspace characterized by C.
  3. Apply Negated Conceptor: Each word vector is multiplied by the negated conceptor (-C * vw). This effectively "softly" dampens the directions in the word vector space that are most influenced by word frequencies, as identified by the conceptor.

This method is purely unsupervised, requiring no external linguistic databases or corpora for training. The "soft" gating mechanism is a key differentiator from the "hard" gating of the ABTT method.

Datasets

The paper evaluates its method on several datasets:

  • Intrinsic Lexical Evaluation Tasks:
    • Word Similarity: RG65, WordSim-353 (WS), rare-words (RW), MEN, MTurk, SimLex-999, SimVerb-3500.
  • Semantic Textual Similarity (STS): SemEval STS tasks (2012-2015) and the SICK dataset.
  • Concept Categorization: ESSLLI 2008, Almuhareb-Poesio (AP), and the BM dataset.
  • Downstream NLP Task:
    • Neural Belief Tracker (NBT): A dialogue state tracking model trained on the Wizard-of-Oz (WOZ) 2.0 dataset.

The paper uses pre-trained Google News Word2Vec and Common Crawl GloVe word embeddings for experiments.

Key Insights and Ideas

  • Conceptors as Soft Gates: The paper frames conceptors as a way to perform "soft" filtering of word vector features, contrasting with the "hard" filtering of methods like ABTT. This allows for more nuanced control over noise reduction.
  • Unsupervised Noise Reduction: The method effectively identifies and suppresses features correlated with word frequency without needing external linguistic resources, making it broadly applicable.
  • Spectral Encode-Decode Framework: Both CN and ABTT can be unified under a "spectral encode-decode" interpretation, involving PC encoding, variance gating, and PC decoding. CN's novelty lies in its variance gating step.
  • SimLex and SimVerb Importance: The paper highlights the improved performance on SimLex and SimVerb benchmarks, which are designed to distinguish genuine word similarity from mere association, suggesting CN better captures nuanced semantic relationships.
  • FastText Resilience: CN and ABTT do not show significant gains on FastText embeddings, possibly because FastText's subword information already mitigates word frequency effects.

Key Results

  • Word Similarity: CN consistently outperforms original word embeddings and ABTT across most word similarity benchmarks, with particularly impressive gains on SimLex-999 (up to 18.86% improvement) and SimVerb-3500 (up to 28.34% improvement) compared to original embeddings.
  • Semantic Textual Similarity (STS): CN also consistently outperforms baselines on STS tasks, demonstrating its effectiveness in capturing sentence-level semantic similarity.
  • Concept Categorization: For smaller datasets, CN and ABTT perform similarly to original embeddings. However, as datasets become larger and more challenging, CN shows superior performance.
  • Neural Belief Tracker (NBT): CN-processed word vectors lead to improved goal accuracy in the NBT dialogue state tracking task, especially when training data is limited, suggesting that good initialization matters.

Limitations and Future Work

  • Computational Cost: Estimating the conceptor involves a matrix product that can be computationally intensive for very large vocabularies. The paper mitigates this by using a subset of words for estimation, which surprisingly also improves performance.
  • Hyperparameter Tuning: The aperture parameter a in the conceptor formulation governs the "sharpness" of the suppression. While the paper uses a fixed value (a=2) for experiments, analytical optimization methods exist, and their connection to the post-processing scheme is a future research direction.
  • FastText Performance: The lack of significant gains on FastText embeddings suggests further investigation into how subword information interacts with post-processing techniques.
  • Future Work: The authors envision using other logical operations of conceptors (AND, OR, NOT) for tasks like inducing bi-lingual word vectors, studying polysemous words, and analyzing graded lexical entailment.

Summary updated

Abstract

Word vectors are at the core of many natural language processing tasks. Recently, there has been interest in post-processing word vectors to enrich their semantic information. In this paper, we introduce a novel word vector post-processing technique based on matrix conceptors (Jaeger 2014), a family of regularized identity maps. More concretely, we propose to use conceptors to suppress those latent features of word vectors having high variances. The proposed method is purely unsupervised: it does not rely on any corpus or external linguistic database. We evaluate the post-processed word vectors on a battery of intrinsic lexical evaluation tasks, showing that the proposed method consistently outperforms existing state-of-the-art alternatives. We also show that post-processed word vectors can be used for the downstream natural language processing task of dialogue state tracking, yielding improved results in different dialogue domains.
BibTeX
@inproceedings{liu2019unsupervised,
  author = {Tianlin Liu and Lyle Ungar and João Sedoc},
  title = {{Unsupervised Post-Processing of Word Vectors via Conceptor Negation.}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33016778},
}
Figure