Interpretable contrastive word mover's embedding

Ruijie Jiang, Julia Gouvea, Eric L. Miller, David Hammer, Shuchin AeronarXiv 2021 · OpenReview

Abstract

This paper shows that a popular approach to the supervised embedding of documents for classification, namely, contrastive Word Mover's Embedding, can be significantly enhanced by adding interpretability. This interpretability is achieved by incorporating a clustering promoting mechanism into the contrastive loss. On several public datasets, we show that our method improves significantly upon existing baselines while providing interpretation to the clusters via identifying a set of keywords that are the most representative of a particular class. Our approach was motivated in part by the need to develop Natural Language Processing (NLP) methods for the novel problem of assessing student work for scientific writing and thinking - a problem that is central to the area of (educational) Learning Sciences (LS). In this context, we show that our approach leads to a meaningful assessment of the student work related to lab reports from a biology class and can help LS researchers gain insights into student understanding and assess evidence of scientific thought processes.

Introduction

In the field of Natural Language Processing (NLP), document embeddings serve as the foundation for various downstream tasks, most notably classification. However, high-performing embedding models often function as "black boxes," providing accurate predictions without explaining the underlying semantic drivers. This lack of transparency is a significant barrier in domains such as the Learning Sciences (LS), where educators require actionable insights into student reasoning rather than just a numerical score.

This paper, presented at NeurIPS 2021, addresses this challenge by introducing an interpretable version of Contrastive Word Mover’s Embedding. By integrating learnable "anchors" into a framework based on Word Mover’s Distance (WMD), the authors enable simultaneous high-accuracy classification and the identification of class-defining keywords.

Methodology: Anchor-Based Contrastive Learning

The proposed method builds upon Word Mover’s Distance (WMD), which measures the minimum "cost" to transform the word distribution of one document into another within the Wasserstein space. While effective, traditional WMD-based classification often relies on K-Nearest Neighbors (KNN), which is computationally expensive ($O(M)$ where $M$ is the training set size).

The authors introduce two primary innovations to overcome these limitations:

  1. Learnable Class Anchors: Instead of comparing documents to every other document in the training set, the model learns a set of "anchors" ($C^{(y)}$) that represent the prototypical distribution of each class.
  2. Contrastive Loss with Anchors: The model utilizes a contrastive loss (such as Triplet or InfoNCE) to learn a transformation matrix ($A$). This matrix projects word embeddings into a space where documents are pulled toward their respective class anchors and pushed away from others.
Conceptual comparison between S-WMD and the proposed Anchor-based WMD framework.

This anchor-based approach reduces inference complexity to $O(1)$ relative to the training set size, as a new document only needs to be compared to a fixed number of class anchors.

Interpretability and Keyword Extraction

A unique advantage of this framework is that anchors and word embeddings exist in the same geometric space. This allows the model to calculate an "importance value" for any word based on its proximity to a specific class anchor relative to others. A word is deemed representative of a class if it is close to that class's anchor but distant from all other anchors.

Class-specific word clouds generated by the model for the BBCSPORT dataset.

As shown in the word clouds above, the model successfully identifies highly relevant keywords for different sports categories. For instance, in the "Cricket" category, the identified keywords were not only semantically relevant but also statistically significant, with the top 30 words appearing almost exclusively within Cricket-related articles.

Experimental Results and Educational Application

The model was evaluated on eight standard NLP benchmarks (e.g., BBCSPORT, REUTERS, AMAZON) and a specialized dataset of 146 biology lab reports.

Performance Benchmarks

The anchor-based method outperformed existing baselines, including Supervised-WMD (S-WMD), on 7 out of 8 public datasets. On the BBCSPORT dataset, the error rate was reduced to 1.7%, a significant improvement over the 4.6% error rate of standard WMD.

Insights into Scientific Writing

In the context of Learning Sciences, the model analyzed student lab reports scored according to the SOLO taxonomy. The results revealed deep qualitative insights:

  • High-Scoring Reports: These were characterized by adverbs such as predominantly, largely, and disproportionately. These "hedging" words indicate sophisticated scientific thinking, where students make qualified, probabilistic claims rather than absolute ones.
  • Low-Scoring Reports: These focused on procedural terminology (e.g., standardize, method, procedure), suggesting a focus on the "what" of the experiment rather than the "why" or the analytical implications.

Conclusion

By combining the geometric strengths of Word Mover’s Distance with a contrastive anchor-based learning objective, this research provides a powerful tool for interpretable document classification. While the model requires more parameters than simpler supervised methods—which can be a challenge for very small datasets—its ability to surface the "why" behind a model's decision makes it an invaluable asset for qualitative research and educational assessment.

Summary updated

Abstract

This paper shows that a popular approach to the supervised embedding of documents for classification, namely, contrastive Word Mover's Embedding, can be significantly enhanced by adding interpretability. This interpretability is achieved by incorporating a clustering promoting mechanism into the contrastive loss. On several public datasets, we show that our method improves significantly upon existing baselines while providing interpretation to the clusters via identifying a set of keywords that are the most representative of a particular class. Our approach was motivated in part by the need to develop Natural Language Processing (NLP) methods for the \textit{novel problem of assessing student work for scientific writing and thinking} - a problem that is central to the area of (educational) Learning Sciences (LS). In this context, we show that our approach leads to a meaningful assessment of the student work related to lab reports from a biology class and can help LS researchers gain insights into student understanding and assess evidence of scientific thought processes.
BibTeX
@misc{jiang2021interpretable,
  author = {Ruijie Jiang and Julia Gouvea and Eric L. Miller and David Hammer and Shuchin Aeron},
  title = {{Interpretable contrastive word mover's embedding}},
  year = {2021},
  eprint = {2111.01023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.01023},
}
Figure