Abstract
Word embeddings are useful for a wide variety of tasks, but they lack interpretability. By rotating word spaces, interpretable dimensions can be identified while preserving the information contained in the embeddings without any loss. In this work, we investigate three methods for making word spaces interpretable by rotation: Densifier (Rothe et al., 2016), linear SVMs and DensRay, a new method we propose. In contrast to Densifier, DensRay can be computed in closed form, is hyperparameter-free and thus more robust than Densifier. We evaluate the three methods on lexicon induction and set-based word analogy. In addition we provide qualitative insights as to how interpretable word spaces can be used for removing gender bias from embeddings.
Introduction
Standard word embeddings, such as Word2vec or FastText, are often treated as "black boxes." While they capture rich semantic relationships, they represent words as dense vectors where individual dimensions lack inherent meaning. This lack of interpretability is a significant hurdle for tasks requiring transparency, such as identifying dimensions for sentiment, concreteness, or gender.
Previous work, specifically the Densifier method, attempted to solve this by learning an orthogonal transformation (rotation) to align linguistic features with specific dimensions. However, Densifier relies on iterative optimization (Stochastic Gradient Descent), which can be unstable and requires extensive hyperparameter tuning. This paper introduces DensRay, a robust, analytical alternative designed to "unlock" these interpretable dimensions without losing the original information contained in the embeddings.
Methodology
The study evaluates three primary approaches to creating interpretable word spaces:
- DensRay: The core contribution of this work. By modifying the Densifier objective function to use squared Euclidean norms, the authors transform the optimization problem into a Rayleigh quotient maximization. This allows for a closed-form, analytical solution via eigendecomposition, making it hyperparameter-free and computationally efficient.
- Linear SVMs/SVRs: The authors explore the effectiveness of simple linear classifiers, finding that the normal vector of an SVM hyperplane can serve as a highly effective interpretable dimension.
- IntCos (Interpretable Cosine): A novel method for set-based word analogies that leverages the interpretable dimensions identified by DensRay or SVMs to improve performance on complex analogy tasks.
Results and Discussion
Performance and Stability
Quantitative results highlight the trade-offs between the methods. In lexicon induction, linear SVMs achieved the highest Macro Mean Kendall’s $\tau$ (0.618), compared to approximately 0.58 for both DensRay and Densifier. This suggests that for identifying a single interpretable dimension, simple linear classifiers are a surprisingly strong baseline.
Crucially, the analytical nature of DensRay provides a major advantage in terms of stability. As shown in the figure above, Densifier (green) exhibits significant instability—manifested as spikes in standard deviation—at certain training data sizes. DensRay (red) and SVMs (black), being closed-form or more robustly optimized, maintain consistent performance. This makes DensRay a more reliable choice for practical applications where hyperparameter tuning is undesirable.
Qualitative Analysis: Gender Debiasing
The ability to isolate specific linguistic dimensions also has implications for algorithmic fairness. By identifying a "gender dimension" using DensRay, the authors demonstrate that one can create a "complement space" by removing this dimension.
Qualitatively, this process moves biased occupation vectors, such as "housekeeper" or "skipper," toward a neutral position between the "man" and "woman" poles in the embedding space. This highlights the potential of interpretable rotations as a tool for auditing and mitigating bias in NLP models.
Conclusion
This work demonstrates that interpretable dimensions can be extracted from dense word embeddings without loss of information through orthogonal rotations. DensRay provides a stable, hyperparameter-free analytical solution, while linear SVMs serve as a powerful baseline for single-dimension identification. Future research will explore higher-dimensional interpretable subspaces and more rigorous quantitative benchmarks for bias removal.
Summary updated