Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base

Vanda Balogh, Gábor Berend, Dimitrios I. Diochnos, György TuránAAAI 2020 · DOI

Abstract

While dense word embeddings like GloVe and Word2Vec have revolutionized natural language processing, their individual dimensions remain largely uninterpretable. This paper explores sparse word embeddings, where words are represented by a small number of "active" dimensions, hypothesized to correspond to semantic "atoms." By leveraging ConceptNet, a massive commonsense knowledge base, the authors provide a computational framework to decode these dimensions. The study demonstrates that sparse dimensions align with human-interpretable concepts and that words grouped within these dimensions exhibit significantly higher semantic coherence than random sets.

Introduction

The "black box" nature of standard dense word embeddings poses a challenge for explainable AI. Sparse word embeddings offer a potential solution by ensuring that only a few dimensions are non-zero for any given word. The goal is for each dimension to represent a specific, interpretable concept (e.g., "politics," "fruit," or "geometry"). However, evaluating the success of this interpretability has historically relied on subjective human judgment. This research introduces a principled approach using ConceptNet to provide a quantitative, symbolic audit of the semantic content within sparse embedding spaces.

Methodology

The authors evaluate three distinct methods for generating sparse embeddings from pre-trained 300-dimensional GloVe vectors:

  1. Dictionary Learning-Based Sparse Coding (DLSC): Optimizes for reconstruction accuracy with a sparsity constraint.
  2. k-means Clustering: Uses cluster centroids as basis vectors.
  3. Greedy Maximization for Pairwise Orthogonality (GMPO): A novel approach that selects actual vocabulary words as basis vectors, prioritizing those that are nearly orthogonal to maximize the distinctness of semantic atoms.

Semantic Alignment via ConceptNet

To interpret these dimensions, the authors use Normalized Positive Pointwise Mutual Information (NPPMI) to map each embedding dimension to the most relevant concept in ConceptNet.

Figure 1: Conceptual mapping between words, sparse basis dimensions, and semantic properties derived from a knowledge base.

To measure coherence, the authors treat ConceptNet as a graph and apply a spreading activation algorithm. They compare the average path length between "top" words (those with high coefficients in a dimension) and "bottom" words (those with zero coefficients). Shorter path lengths indicate higher semantic relatedness.

Key Insights and Results

The research yields several critical findings regarding the nature of sparse representations:

  • Sparsity and Interpretability: The results confirm that sparsity and non-negativity constraints naturally lead to dimensions that align with symbolic knowledge.
  • Semantic Coherence: In 66% to 73% of analyzed dimensions, "top" words had significantly shorter average path lengths (~1.8 to 2.0) in ConceptNet compared to random words (~3.4 to 4.0). This provides empirical evidence that the mathematical clusters correspond to real-world semantic groups.
  • Superiority of Orthogonality: The GMPO method, by explicitly seeking orthogonal basis vectors, produced more distinct and interpretable semantic atoms than traditional k-means.
Figure 3: Distribution of inner products for active versus inactive semantic atoms, illustrating the statistical separation achieved through sparsification.
  • Handling Polysemy: Sparse embeddings effectively manage polysemous words. A word like "court" can simultaneously activate dimensions related to "law" and "sports," a feat difficult for single-vector dense representations.

Conclusion and Future Work

This work bridges the gap between statistical vector spaces and symbolic knowledge. By using ConceptNet as a "decoder," the authors demonstrate that sparse embedding dimensions are not merely mathematical artifacts but represent coherent semantic atoms.

Future research will address the inherent noise in crowd-sourced knowledge bases like ConceptNet and explore more nuanced graph metrics, such as edge labels (e.g., distinguishing "is-a" from "part-of"). Additionally, the authors aim to apply these evaluation tools to dense embeddings to uncover any latent interpretability within standard models.

Summary updated

BibTeX
@inproceedings{balogh2020understanding,
  author = {Vanda Balogh and Gábor Berend and Dimitrios I. Diochnos and György Turán},
  title = {{Understanding the Semantic Content of Sparse Word Embeddings Using a Commonsense Knowledge Base}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6235},
}
Figure