Unsupervised Post-processing of Word Vectors via Conceptor Negation

Tianlin Liu, Lyle Ungar, João Sedoc

TL;DR

This paper introduces Conceptor Negation (CN), a simple yet powerful unsupervised post-processing technique that improves the semantic quality of word vectors. By using matrix conceptors to "softly" suppress high-variance features—which typically encode word frequency noise rather than meaning—CN consistently outperforms existing state-of-the-art methods across various lexical and downstream NLP tasks.

What Problem Does This Paper Address?

Word vectors like Word2Vec and GloVe are fundamental to modern NLP, but they aren't perfect. A known issue is that their leading principal components (the directions of highest variance) tend to encode word frequency rather than semantic meaning. Because word frequency is often unrelated to lexical semantics, these components act as "noise."

Previous research proposed the All-But-The-Top (ABTT) method, which simply deletes the top $d$ principal components. However, ABTT is a "hard" binary operation: a feature is either completely removed or kept entirely intact. This can lead to a "Goldilocks" problem—removing too little noise or losing too much useful information. The authors seek a more nuanced, "softer" way to filter this noise without requiring external labels or massive retraining.

Methods

The authors propose Conceptor Negation (CN), based on the theory of matrix conceptors—a family of regularized identity maps.

  1. Conceptor Computation: They first compute a conceptor matrix $C$ that characterizes the distribution of the word vectors. This matrix acts as a soft projection onto the high-variance subspace.
  2. Negation: They then use the Boolean "NOT" operation for conceptors ($\neg C = I - C$).
  3. Soft Gating: Applying this negated conceptor to the word vectors dampens the high-variance (frequency-heavy) directions.

Unlike ABTT's hard cutoff, CN performs soft gating. As shown in Figure 1, while ABTT sets the first few components to zero and keeps the rest at one, CN uses a smooth curve to gradually suppress components based on their variance.

Figure 1: The variance gating coefficients of ABTT and CN for Word2Vec. Hyper-parameters: d = 3 for ABTT and α = 2 for CN.

Datasets

The authors evaluated their method on a comprehensive suite of benchmarks:

  • Word Similarity: Seven datasets including RG65, WordSim-353, Rare-words (RW), MEN, MTurk, and the challenging SimLex-999 and SimVerb-3500.
  • Semantic Textual Similarity (STS): Tasks from SemEval 2012–2015 and the SICK dataset.
  • Concept Categorization: ESSLLI 2008, Almuhareb-Poesio (AP), and the BM dataset.
  • Downstream Task: Dialogue State Tracking using the Neural Belief Tracker (NBT) on the Wizard-of-Oz (WOZ) 2.0 dataset.

Key Insights and Ideas

  • Soft vs. Hard Gating: The core "aha!" moment is that semantics and noise aren't perfectly separated into discrete principal components. A soft filter (CN) preserves more semantic nuance than a hard axe (ABTT).
  • Unifying Framework: The paper shows that both ABTT and CN can be viewed as "spectral encode-decode processes." This provides a theoretical bridge between neural word embeddings and older techniques like Eigenvalue Weighting used in Pointwise Mutual Information (PMI) matrices.
  • Efficiency Trick: The authors found that estimating the conceptor matrix using only common words (appearing >200 times in Wikipedia) not only speeds up computation but actually improves performance, likely because vectors for rare words are noisier.
  • Fasttext Exception: Interestingly, CN doesn't help Fasttext vectors. The authors hypothesize this is because Fasttext’s use of subword information already inherently suppresses frequency-based noise during training.

Key Results

CN consistently improved performance over both original vectors and the ABTT baseline.

  • Word Similarity: CN achieved significant gains, particularly on SimLex-999 and SimVerb-3500, which are designed to distinguish true similarity from simple association (e.g., "coffee" and "cup" are associated but not similar). On SimLex-999, CN reached performance levels comparable to supervised fine-tuning methods (see Table 2 in the paper).
Table 1: Word similarity results. CN (bold) outperforms the original and ABTT vectors in nearly every category.
  • Dialogue State Tracking: In the downstream NBT task, using CN-processed vectors as initialization led to better "goal accuracy" across different domains like food and price range, showing that these intrinsic improvements translate to real-world utility.
Table 5: Goal accuracy for dialogue state tracking. CN-processed vectors provide a superior initialization for the neural tracker.

Limitations and Future Work

  • Hyperparameter Tuning: The "aperture" $\alpha$ is a hyperparameter that controls the sharpness of the filter. While the authors used a fixed value ($\alpha=2$) for most experiments, they note that analytical optimization methods for $\alpha$ exist and could be integrated.
  • Task Specificity: While CN helps in many areas, it yields equivalent results to ABTT in tasks like text classification, where the vectors are only perturbed slightly.
  • Logical Operations: The authors suggest that other conceptor operations, like AND ($\wedge$) for bilingual mapping or OR ($\vee$) for handling polysemy (words with multiple meanings), are promising avenues for future research.

Summary updated

Abstract

Word vectors are at the core of many natural language processing tasks. Recently, there has been interest in post-processing word vectors to enrich their semantic information. In this paper, we introduce a novel word vector post-processing technique based on matrix conceptors (Jaeger2014), a family of regularized identity maps. More concretely, we propose to use conceptors to suppress those latent features of word vectors having high variances. The proposed method is purely unsupervised: it does not rely on any corpus or external linguistic database. We evaluate the post-processed word vectors on a battery of intrinsic lexical evaluation tasks, showing that the proposed method consistently outperforms existing state-of-the-art alternatives. We also show that post-processed word vectors can be used for the downstream natural language processing task of dialogue state tracking, yielding improved results in different dialogue domains.
BibTeX
@inproceedings{liu2019unsupervised,
  author = {Tianlin Liu and Lyle Ungar and João Sedoc},
  title = {{Unsupervised Post-processing of Word Vectors via Conceptor Negation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33016778},
}
Figure