TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora

Denis Newman-Griffis, Venkatesh Sivaraman, Adam Perer, Eric Fosler-Lussier, Harry Hochheiser

Introduction

Word embeddings have become a cornerstone of modern Natural Language Processing (NLP), transforming words into high-dimensional vectors that represent semantic relationships. While typically used as static features for downstream models, embeddings also hold immense potential as tools for corpus-level discovery. TextEssence is an interactive system designed to bridge the gap between raw text data and semantic insights, allowing researchers to visualize and compare how word meanings shift across different text collections or time periods.

The Challenge: Comparison and Instability

The development of TextEssence addresses two primary limitations in current embedding analysis:

  1. The Comparison Gap: Existing tools often focus on a single corpus. There is a lack of interactive systems specifically designed to highlight semantic drift—the change in a word's "vibe" or meaning—when moving between datasets, such as scientific literature from different years.
  2. The Instability Problem: Embedding training is inherently stochastic. Small datasets can introduce "noise," where a word's position in the vector space changes due to algorithmic randomness rather than actual semantic shifts. This makes it difficult to distinguish meaningful drift from training artifacts.

Methodology: The TextEssence System

TextEssence provides a web-based interface with three integrated modes for exploring semantic shifts: Browse, Inspect, and Compare. The system uses Procrustes transformation to align embedding spaces from different corpora, ensuring that points move smoothly during transitions to highlight genuine changes.

The TextEssence pipeline and its three interactive modes: Browse, Inspect, and Compare.

Embedding Confidence (EC@k)

To mitigate the instability problem, the authors introduced a novel metric called Embedding Confidence (EC@k). Instead of relying on a single training run, the system generates 10 independent replicates of embeddings for each corpus. The confidence score for a term is calculated based on the consistency of its nearest neighbors across these replicates.

The Overlap Analysis process for calculating Embedding Confidence by comparing nearest neighbors across multiple replicates.

By filtering for "stable" words (typically using a threshold of 0.5), TextEssence ensures that the visualized shifts are statistically significant and not merely results of training noise.

Case Study: Semantic Drift in COVID-19 Research

The system was evaluated using the CORD-19 (COVID-19 Open Research Dataset), analyzing monthly snapshots of scientific literature from March to October 2020. This period saw a rapid evolution in the understanding of the virus, making it an ideal testbed for tracking semantic drift.

Key Findings

  • Anosmia (Loss of Smell): Early in the pandemic, "anosmia" was associated with general symptoms like "pharyngeal pain." By October, its neighborhood shifted to include "vomiting" and "nausea," reflecting its recognition as a hallmark diagnostic symptom within a broader clinical syndrome.
  • Dexamethasone: This steroid's semantic neighborhood initially centered on "traumatic brain injury." Following the landmark RECOVERY trial results, it shifted dramatically toward other COVID-19 treatments such as "Prednisone" and "Ruxolitinib."

Conclusion

TextEssence demonstrates that static embeddings (e.g., word2vec) remain highly effective for corpus-level analysis, as they capture aggregate patterns more efficiently than resource-intensive contextualized models. By treating embedding instability as a measurable signal rather than a bug, TextEssence provides a robust framework for discovering how language evolves in response to real-world events.

Summary updated

BibTeX
@inproceedings{newmangriffis2021textessence,
  author = {Denis Newman-Griffis and Venkatesh Sivaraman and Adam Perer and Eric Fosler-Lussier and Harry Hochheiser},
  title = {{TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-demos.13},
}
Figure