Identification, Interpretability, and Bayesian Word Embeddings

Adam LauretigNAACL 2019

TL;DR

This paper introduces Bayesian Word Embeddings with Automatic Relevance Determination, a novel method for creating interpretable word embeddings that can be directly used in social science regressions. By addressing the identification and interpretability issues of traditional word embeddings, this approach allows for more robust analysis of textual data, as demonstrated by its application to historical political rhetoric.

What Problem Does This Paper Address?

Social scientists increasingly use word embeddings, a technique from natural language processing, to analyze text and measure concepts like ideology, bias, and affinity. However, these embeddings present two major challenges for social science research: they are not "identified" (meaning multiple interpretations can explain the same data) and their dimensions are not directly interpretable. This makes it difficult to use them in standard statistical regression models, where understanding the meaning and scale of coefficients is crucial. The paper aims to bridge this gap by making word embeddings both identifiable and interpretable for social science applications.

Methods

The core contribution of this paper is the development of Bayesian Word Embeddings with Automatic Relevance Determination (ARD) priors. This approach treats word embeddings as a Bayesian latent variable model.

  • Bayesian Framework: By casting the problem in a Bayesian framework, the authors can quantify uncertainty in parameter estimates and incorporate prior information.
  • Automatic Relevance Determination (ARD): ARD priors are introduced to relax the assumption that all embedding dimensions are equally important. This is achieved by placing a gamma-distributed scalar on the diagonal of the covariance matrix for each dimension, effectively penalizing unnecessary dimensions and improving model fit.
  • Identification and Interpretability: To address the identification problem, the paper draws on techniques from ideal point modeling. It proposes a method to anchor dimensions by specifying theoretically motivated endpoints (e.g., "international" and "domestic") and then using cosine similarity to find opposite anchors for other dimensions. An affine transformation is then applied to scale these dimensions, making them directly interpretable and usable as covariates in regression models.
  • Variational Inference: The model is estimated using variational inference, a common technique for approximating complex posterior distributions in Bayesian models.

Datasets

The paper uses two primary datasets for its analysis:

  • Presidential Inaugural Addresses: A corpus of 58 speeches, used to investigate shifts in internationalism in US foreign policy rhetoric before and after 1945.
  • Declassified Diplomatic Documents (Foreign Relations of the United States - FRUS): A collection of declassified diplomatic documents, used to examine the relationship between elite rhetoric and US foreign policy behavior, specifically bellicosity.

Key Insights and Ideas

  • Beyond Document-Level Analysis: Traditional text analysis methods often focus on documents as units. This work enables inference at the word level, uncovering patterns that document-based topic models might miss.
  • The Power of Anchoring: Simply anchoring an embedding on a single word is less informative than anchoring it on a pair of opposing concepts (e.g., "war" vs. "peace"). This allows for the creation of meaningful scales (like "bellicosity") that can be directly interpreted.
  • Challenging Conventional Wisdom: The analysis of inaugural addresses suggests that US rhetoric became less internationalist after World War II, contrary to some theoretical expectations. This highlights the potential of the method to reveal nuanced historical shifts.
  • Elite Deliberation Matters: The study on diplomatic documents indicates that bellicose rhetoric in elite discussions is correlated with increased hostile foreign policy actions, suggesting that these deliberations are not mere "cheap talk" but have tangible consequences.

Key Results

  • Inaugural Addresses: A statistically significant shift was found, indicating that pre-1945 inaugural addresses were more "internationalist" than post-1945 addresses. This finding held even when excluding the Roosevelt administration and focusing on addresses before 1932.
  • Diplomatic Documents: A positive and statistically significant relationship was found between bi-weekly bellicosity scores (derived from the "war-peace" dimension) and US-initiated material conflict events. This suggests that increased bellicosity in elite deliberations precedes an increase in hostile actions. The results remained consistent even after removing outliers.

Limitations and Future Work

  • Computational Scalability: While variational inference is used, scaling to massive corpora is still a challenge. The authors suggest exploring stochastic variational inference for future work.
  • Incorporating Metadata: Future research could incorporate document-level metadata directly into the embedding estimation process, allowing embeddings to vary based on document-specific attributes.
  • Causal Claims: The authors propose tying the anchoring approach to emerging literature on making causal claims from text, aiming to identify appropriate linguistic counterfactuals.
  • Generalizability: While the paper demonstrates the utility of the method on two specific corpora, further validation across a wider range of social science applications would be beneficial.

Summary updated

BibTeX
@inproceedings{lauretig2019identification,
  author = {Adam Lauretig},
  title = {{Identification, Interpretability, and Bayesian Word Embeddings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure