Mixed Membership Word Embeddings for Computational Social Science

James R. FouldsAISTATS 2018

Presented at AISTATS 2018, this research bridges the gap between the high-performance but "black box" nature of neural word embeddings and the interpretable structure of probabilistic topic modeling. The authors introduce a method to generate high-quality, interpretable word representations even when working with the limited datasets common in the social sciences.

The Interpretability Gap in NLP

Modern Natural Language Processing (NLP) relies heavily on word embeddings like word2vec (skip-gram). While effective, these models present two significant hurdles for Computational Social Science (CSS):

  1. Data Hunger: Standard embeddings typically require massive, internet-scale corpora. Social scientists often work with niche, high-value datasets—such as a century of political speeches—that are too small for traditional neural models to learn effectively.
  2. Lack of Interpretability: Standard embeddings are dense numerical vectors. They lack the inherent semantic transparency of topic models, making it difficult for researchers to explain why words cluster together.

Furthermore, using pre-trained "big data" vectors on specific domains often fails because context changes. In a general news corpus, "learning" might relate to education; in a machine learning corpus, it relates to optimization.

Mixed Membership Skip-Gram (MMSG)

The core innovation is the Mixed Membership Skip-Gram (MMSG). This model reinterprets the skip-gram architecture through the lens of topic modeling, specifically drawing inspiration from Latent Dirichlet Allocation (LDA).

Conceptual illustration of mixed membership: the word "learning" as a distribution over distinct topics like machine learning, classroom learning, and life learning.

Key Methodological Shifts

  • Parameter Sharing: Instead of assigning every word a unique, independent vector, words are represented as a distribution over a shared set of $K$ "topic" vectors. This allows the model to share statistical strength across the vocabulary, which is crucial for small datasets.
  • Generative Process: To predict a context word, the model selects a topic based on the input word's specific distribution and then draws the context word using that topic's vector.
  • Efficient Training: To handle the computational cost, the authors utilize an annealed Metropolis-Hastings-Walker (MHW) sampler for sublinear-time topic assignment and Noise-Contrastive Estimation (NCE) to learn vectors without normalizing over the entire dictionary.

Evaluation and Key Insights

The model was evaluated on several corpora of sociopolitical and scientific interest, including NIPS articles (1987–1999), U.S. State of the Union (SOTU) addresses, and the works of W.E.B. Du Bois.

Domain Specificity and Small Data

Generic vectors (like Google News) were consistently outperformed by domain-specific MMSG models trained on much smaller datasets. In predicting context words (Mean Reciprocal Rank), the MMSG variant outperformed standard skip-grams by up to 63% on the NIPS dataset.

Handling Polysemy

Because MMSG uses mixed membership, it naturally handles words with multiple meanings. A word like "Bayesian" can be decomposed into specific topics like "MCMC" or "Neural Networks" rather than being forced into a single point in vector space.

Social Science Utility

For CSS researchers, the model's ability to recover coherent topics is its greatest strength. Using t-SNE visualizations of SOTU addresses, the model revealed clear temporal trajectories and thematic shifts, such as the emergence of the "War on Terror" topic in the early 21st century.

t-SNE visualization of State of the Union addresses, demonstrating how the model captures temporal trajectories and thematic clustering over centuries.

Conclusion

MMSG provides a powerful tool for researchers who need the predictive power of embeddings but require the interpretability of topic models. By excelling in small-to-medium data regimes, it opens new doors for rigorous computational analysis of historical and specialized texts. Future work may explore how this increased representational power scales to massive datasets and its potential for identifying algorithmic bias in NLP.

Summary updated

BibTeX
@inproceedings{foulds2018mixed,
  author = {James R. Foulds},
  title = {{Mixed Membership Word Embeddings for Computational Social Science}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2018},
}
Figure