Analytical Methods for Interpretable Ultradense Word Embeddings

Philipp Dufter, Hinrich Schütze

Philipp Dufter, Hinrich Schütze
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019.

Introduction

Word embeddings like Word2Vec and FastText are powerful tools in NLP, but they often function as "black boxes." Information regarding specific linguistic features—such as sentiment, concreteness, or gender—is typically entangled across hundreds of dimensions, making them difficult for humans to interpret or manipulate directly.

Previous research introduced the Densifier method, which learns an orthogonal transformation (a rotation) to align specific features with designated axes. However, Densifier relies on stochastic gradient descent (SGD), which can be unstable, sensitive to hyperparameters, and prone to high variance on smaller datasets. This paper introduces DensRay, an analytical, closed-form solution that provides a more robust and mathematically grounded approach to creating interpretable ultradense embeddings.

Methodology

The researchers propose two primary alternatives to the iterative optimization used in previous work:

  1. DensRay (Density Rayleigh): By reformulating the original objective function using squared Euclidean norms, the authors derive a solution using the Rayleigh quotient and eigendecomposition. This closed-form solution is hyperparameter-free and significantly more stable than SGD-based methods.
  2. Linear SVMs/SVRs: Surprisingly, the authors found that simple linear Support Vector Machines (or Regressions) are highly effective. By treating the normal vector of the SVM hyperplane as the "interpretable dimension," they can extract features with high accuracy.
  3. IntCos (Interpretable Cosine): A novel method for solving word analogies (e.g., "man is to king as woman is to X") by leveraging these newly defined interpretable dimensions.

Crucially, these methods utilize orthogonal transformations, ensuring that the original geometry and distances within the embedding space are preserved—only the "camera angle" changes to highlight specific features.

Key Insights: Stability and Performance

A major contribution of this work is the demonstration of DensRay's robustness. Because it is an analytical solution, it avoids the instability inherent in iterative optimization. As shown in Figure 1, Densifier's performance variance (standard deviation) can spike dramatically on medium-sized training sets, whereas DensRay remains rock-solid.

Performance and Stability: Mean performance (top) and standard deviation (bottom) across 40 samples. DensRay provides stability where Densifier (Dens.) fails, while SVMs offer superior mean performance.

Furthermore, the study highlights the "Power of Simplicity." In lexicon induction tasks, simple linear SVMs consistently outperformed both Densifier and DensRay. This suggests that for identifying a single interpretable direction, standard classification tools are already highly optimized for the task.

Applications and Results

The authors evaluated their methods across multiple languages and tasks, including sentiment analysis and gender debiasing:

  • Lexicon Induction: SVMs achieved the highest score (0.618 Macro Mean Kendall’s $\tau$), followed by DensRay (0.581) and Densifier (0.580).
  • Surgical Debiasing: By identifying a "gender dimension" via rotation, the authors could remove it to create a debiased subspace. Qualitatively, this moved biased occupation vectors (e.g., "housekeeper" or "skipper") much closer to a neutral position between "man" and "woman" vectors, as seen in Figure 2.
Surgical Debiasing: Occupation vectors in the debiased subspace. Biased terms like "housekeeper" and "skipper" are moved closer to the neutral diagonal between "man" and "woman" vectors.

Conclusion

DensRay offers a mathematically elegant and stable alternative for creating interpretable word embeddings without information loss. While specialized methods like DensRay provide theoretical advantages and stability, the unexpected strength of linear SVMs suggests that standard machine learning tools are often sufficient for identifying interpretable directions in vector spaces. Future work will explore extending these analytical methods to multi-dimensional subspaces and more rigorous quantitative evaluations of debiased spaces.

Summary updated

BibTeX
@inproceedings{dufter2019analytical,
  author = {Philipp Dufter and Hinrich Schütze},
  title = {{Analytical Methods for Interpretable Ultradense Word Embeddings}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=vL4ahBpV4l2},
}
Figure