xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks

Ta-Chung ChiOpenReview 2018 · OpenReview

TL;DR

This paper introduces xSense, a novel model that tackles the interpretability challenge of word embeddings by learning sense-specific sparse representations and generating human-readable textual definitions. By disentangling word senses into distinct dimensions and leveraging context, xSense offers a more transparent understanding of word meanings and achieves state-of-the-art results in definition generation.

What Problem Does This Paper Address?

Word embeddings, while powerful for NLP tasks, are notoriously difficult to interpret. This is due to three main issues: polysemy (a single vector representing multiple meanings), dimension understanding (the meaning of individual dimensions is unclear), and semantic analysis (understanding the overall meaning requires indirect methods like examining nearest neighbors). Existing methods have struggled to address all these issues simultaneously, often sacrificing interpretability for performance or failing to handle polysemy effectively. This paper aims to bridge this gap by providing a model that can not only separate word senses but also explain them through natural language definitions.

Methods

xSense is built upon four key modules:

  • Context Encoder: This module generates a meaningful sentence embedding from the given context. It uses a weighted-sum approach based on Smooth Inverse Frequency (SIF) embeddings, which are inspired by discourse random walk models. This component is not trained, making it more transparent.
  • Sparse Vector Extractor: This module projects the dense word embedding into a high-dimensional sparse vector. This is achieved using a k-sparse autoencoder, which aims to disentangle different senses into distinct dimensions. The idea is that large values in specific dimensions of the sparse vector correspond to particular semantic components.
  • Mask Generator: This is the core of xSense's interpretability. It uses an attention mechanism to identify which dimensions in the sparse vector are most relevant to the target word's sense in the given context. It aligns the sentence embedding with the basis vectors from the sparse representation to generate a sense-specific mask.
  • Definition Decoder: This module, a GRU-based recurrent neural network, takes the context and the generated sense vector (a weighted sum of the most relevant sparse dimensions) to produce a human-readable textual definition for the target word. It uses dual vocabularies to handle common words that might be missing from the primary word embedding vocabulary.

Datasets

The paper introduces a newly collected, large, and high-quality context-definition dataset. This dataset is an improvement over existing ones, providing:

  • Multiple example sentences per definition.
  • Example sentences that consistently contain the target word.
  • Alignment between example sentences and their associated definitions.
  • Corresponding Part-of-Speech (POS) tags for each word sense.

The dataset is significantly larger (approximately 5 times) than the one used by Gadetsky, Yakubovskiy, and Vetrov (2018). They also use this dataset for training and evaluation, along with subsets for "Large," "Small," and "Unseen" test cases to evaluate performance on seen and unseen words/contexts.

Key Insights and Ideas

  • Disentangling Senses via Sparsity: The core idea is that by projecting dense embeddings into a sparse space, different senses of a polysemous word can be isolated into distinct dimensions. This directly addresses the polysemy issue.
  • Contextual Attention for Sense Selection: The mask generator's ability to use context to select relevant dimensions from the sparse representation is crucial. It allows the model to pick out the specific sense of a word that is active in a given situation.
  • Textual Definitions as the Ultimate Explanation: Generating natural language definitions provides a direct and human-understandable explanation of a word's meaning in context, overcoming the "dimension understanding" and "semantic analysis" challenges.
  • The Importance of High-Quality Data: The creation of a richer, more accurate context-definition dataset is highlighted as a significant contribution, enabling better training for definition modeling and related tasks.

Key Results

  • Quantitative Performance: xSense achieves state-of-the-art results on BLEU and ROUGE-L scores across various datasets (Large, Small, Unseen), outperforming several strong baselines. Notably, the "ATS" (Aligned Contexts/Target Word/Sense Vector) variant of xSense shows the best performance.
  • Human Evaluation: In a human evaluation on 200 randomly sampled questions, xSense (specifically the ATS variant) performed best among all candidates, demonstrating superior quality and relevance of its generated definitions.
  • Effectiveness of Context: Models that leverage context (like xSense) generally perform better than those that don't, especially on more challenging test sets.
  • Limitations of Other Approaches: The paper highlights that models like Gadetsky et al.'s, while using masks, still struggle with unseen datasets, suggesting overfitting to training data. Noraset et al.'s model, while generating consistent definitions, doesn't adapt to context.

Limitations and Future Work

  • Unseen Words Challenge: Performance degrades on the "Unseen" dataset, which consists of words not encountered during training. This is attributed to the difficulty of explaining embeddings for unknown words and the fact that these words are often uncommon.
  • Sense Imbalance: For polysemous words, the model can sometimes generate definitions for less frequent senses, suggesting that the imbalance of sense frequencies in the training data can lead to errors.
  • SIF Embedding Propagation: Errors in the SIF sentence embedding can propagate to the generated definitions and nearest neighbors.
  • Future Work: The paper suggests that the created dataset and toolkit can be used for various NLP tasks. The model's ability to disentangle senses and generate definitions opens avenues for further research in explainable AI for NLP.

Summary updated

Abstract

Despite the success achieved on various natural language processing tasks, word embeddings are difficult to interpret due to the dense vector representations. This paper focuses on interpreting the embeddings for various aspects, including sense separation in the vector dimensions and definition generation. Specifically, given a context together with a target word, our algorithm first projects the target word embedding to a high-dimensional sparse vector and picks the specific dimensions that can best explain the semantic meaning of the target word by the encoded contextual information, where the sense of the target word can be indirectly inferred. Finally, our algorithm applies an RNN to generate the textual definition of the target word in the human readable form, which enables direct interpretation of the corresponding word embedding. This paper also introduces a large and high-quality context-definition dataset that consists of sense definitions together with multiple example sentences per polysemous word, which is a valuable resource for definition modeling and word sense disambiguation. The conducted experiments show the superior performance in BLEU score and the human evaluation test.
BibTeX
@misc{chi2018xsense,
  author = {Ta-Chung Chi},
  title = {{xsense: Learning sense-separated sparse representations and textual definitions for explainable word sense networks}},
  year = {2018},
  eprint = {1809.03348},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.03348},
  howpublished = {OpenReview},
}
Figure