Interpretable Word Embeddings via Informative Priors

Miriam Hurtado Bodell, Martin Arvidsson, Måns MagnussonEMNLP 2019 · DOI

TL;DR

This paper introduces a novel method for making word embeddings, the building blocks of many NLP tasks, more interpretable. By using "informative priors" – essentially, pre-defined constraints based on domain knowledge – they can guide the embedding process to create dimensions that directly correspond to specific semantic concepts like gender or sentiment, improving their utility for fields like computational social science.

What Problem Does This Paper Address?

Word embeddings, like Word2Vec or GloVe, have revolutionized Natural Language Processing (NLP) by representing words as dense numerical vectors. This allows computers to understand word relationships and perform complex tasks. However, a major drawback is their "black box" nature: the individual dimensions of these vectors lack clear semantic meaning, making it difficult to understand why certain words are considered similar. This lack of interpretability is a significant barrier for researchers in fields like computational social science and digital humanities, who need to understand the nuances of language and its underlying concepts. Furthermore, existing methods offer limited ways to incorporate specific domain knowledge or research questions directly into the embedding process.

Methods

The core innovation of this paper lies in the use of "informative priors" within a probabilistic framework for word embeddings. Instead of relying solely on unsupervised learning from text, they introduce constraints that guide the embedding process. Specifically, they propose:

  • Standard Basis Prior: This method uses sets of "anchor words" (e.g., "man" and "woman" for a gender dimension) to define specific dimensions. The prior encourages words similar to "man" to have a high value in a particular dimension, and words similar to "woman" to have a low value. They explore "strict" and "weak" versions of this prior, differing in how strongly they enforce these constraints.
  • Truncated Prior: This approach relaxes the assumption that anchor words must have exact locations, instead guiding them to lie on the positive or negative side of a dimension. This is useful when the precise location isn't known or when dealing with concepts that aren't strictly bipolar.
  • Using Neutral Words: They also introduce a way to incorporate "neutral" words (like stop words) that are expected to have little to no bearing on a specific semantic dimension. By pushing these words towards zero in that dimension, they can help isolate the semantic concept of interest.

These priors are integrated into a probabilistic model of word embeddings, allowing for the incorporation of domain knowledge directly during training, rather than as a post-hoc analysis.

Datasets

The authors evaluated their methods on three commonly used English corpora for textual analysis within computational social science:

  • Project Gutenberg: A collection of the top 100 books.
  • Twitter: A sample of tweets.
  • U.S. Congress Speeches (1981-2016): A dataset of political discourse.

The size of these datasets varied from 1.8 million to 40 million tokens after pre-processing. They also used hold-out test words from established sources like the Social Security Administration's top names list and the AFINN dictionary for sentiment analysis to evaluate the interpretability of the learned dimensions.

Key Insights and Ideas

  • Priors unlock interpretability: The fundamental idea that pre-defined semantic constraints (priors) can directly create interpretable dimensions in word embeddings is the paper's main contribution.
  • Anchor words are key: The more anchor words used, the better the model captures the intended semantic dimension, with diminishing returns after a certain point.
  • Flexibility matters: Relaxing strict location assumptions (as in the Truncated Prior) can lead to comparable or even better results, suggesting that perfect knowledge of anchor word positions isn't always necessary.
  • Neutral words help isolate concepts: Guiding neutral words to have zero value in a target dimension can improve the clarity and isolation of that dimension.
  • Semantic change is observable: The dynamic embedding model, combined with informative priors, can reveal temporal shifts in word meanings, as demonstrated by the sentiment trajectories of words related to terror events.

Key Results

The proposed methods generally performed on par with or better than the state-of-the-art (SOTA) methods for capturing predefined semantic dimensions like gender and sentiment.

  • Accuracy: The "weak" Standard Basis Prior and the Truncated Prior showed strong performance, often outperforming SOTA. The accuracy generally increased with the number of anchor words.
  • Gender Dimension: The method successfully captured gender-related concepts, with hold-out words being correctly classified.
  • Sentiment Dimension: The approach proved effective in capturing sentiment, a dimension that is notoriously difficult for standard word embeddings.
  • Semantic Change: The paper demonstrated the ability to track temporal changes in word meaning, showing how words like "September" and "Oklahoma" exhibited significant sentiment shifts around major events.

Limitations and Future Work

  • Concept Isolation: The authors acknowledge that achieving perfect isolation of concepts within a single dimension can be challenging, especially for ambiguous words or when concepts are intertwined. They suggest that future work could involve applying informative priors to multiple dimensions simultaneously to address this.
  • Ambiguous Names: In the gender dimension, some ambiguous names (e.g., "Madison") led to misclassification, highlighting the difficulty of completely disentangling concepts.
  • Data Size: While the method performed well across various corpora, the paper notes that corpus size can influence results, with larger corpora sometimes showing better performance.

The authors suggest that their flexible framework can be extended to other probabilistic word embedding models and applied to more complex scenarios, such as capturing multiple semantic dimensions simultaneously.

Summary updated

BibTeX
@inproceedings{bodell2019interpretable,
  author = {Miriam Hurtado Bodell and Martin Arvidsson and Måns Magnusson},
  title = {{Interpretable Word Embeddings via Informative Priors}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1661},
}
Figure