TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora

Denis Newman-Griffis, Venkatesh Sivaraman, Adam Perer, Eric Fosler-Lussier, Harry Hochheiser

Introduction

Word embeddings are a cornerstone of modern Natural Language Processing (NLP), providing mathematical representations of words based on their linguistic context. However, using these embeddings to compare different text collections—such as scientific literature from different time periods—presents significant challenges. Most existing methods focus on quantitative metrics that indicate that a change occurred, but fail to provide qualitative insights into how meanings shifted. Furthermore, embedding models are notoriously unstable; training the same model twice on the same data can yield different results, making it difficult to distinguish genuine semantic shifts from algorithmic noise.

To address these issues, we present TextEssence, an interactive web-based tool designed for the visual and qualitative exploration of semantic shifts across corpora. TextEssence introduces a novel "embedding confidence" metric to filter out unstable results, ensuring that the shifts observed by researchers are statistically robust.

The TextEssence Interactive Workflow

TextEssence is built with a Python backend and a Svelte frontend, offering three primary modes of exploration to help researchers navigate complex datasets:

  1. Browse: Users can view a 2D map of concepts generated via t-SNE. To ensure meaningful comparisons, the tool uses a Procrustes transformation to align maps from different corpora. Visual "trajectories" show how concepts move between datasets, with line thickness indicating the magnitude of the shift.
  2. Inspect: This mode allows for a deep dive into the "neighborhood" of a specific concept—the words most similar to it—to see how its immediate context evolves over time.
  3. Compare: Researchers can track the relationship between two specific concepts across multiple datasets, identifying when and how their similarity peaks or declines.
The TextEssence interactive workflow: Browse, Inspect, and Compare.

Technical Innovation: Embedding Confidence ($EC@k$)

A key contribution of TextEssence is the Embedding Confidence ($EC@k$) metric. To mitigate the instability of embedding algorithms, the system trains multiple "replicates" (e.g., 10 versions) of the same model. It then measures the overlap of a word's nearest neighbors across these replicates. If a word's neighbors are consistent across all runs, it is assigned high confidence; if they fluctuate, the result is considered unreliable and filtered from the visualization.

Overlap Analysis for calculating Embedding Confidence across model replicates.

By requiring a confidence threshold (typically 0.5), TextEssence ensures that users only analyze stable, reproducible semantic patterns. While modern contextualized models like BERT are popular, TextEssence utilizes static embeddings (e.g., word2vec-style) because they are more resource-efficient for training the necessary replicates and better at capturing aggregate corpus-level patterns.

Case Study: COVID-19 Scientific Literature

We applied TextEssence to the CORD-19 (COVID-19 Open Research Dataset), analyzing monthly snapshots of scientific papers from March to October 2020. To improve conceptual clarity, we used ScispaCy and JET to map text to SNOMED CT medical codes, allowing the tool to treat synonymous terms (e.g., "Anosmia" and "Loss of smell") as single entities.

The analysis revealed several key scientific shifts:

  • Anosmia: Early in the pandemic, its neighbors were general symptoms like "taste" and "pharyngeal pain." By October, it shifted toward systemic COVID-19 symptoms like "nausea," reflecting its emergence as a hallmark diagnostic indicator.
  • Dexamethasone: Initially associated with "traumatic brain injury," this steroid moved into a stable neighborhood of respiratory treatments and antivirals following clinical trials that proved its efficacy for COVID-19.
  • Ruxolitinib: The tool identified a sharp increase in similarity between Dexamethasone and Ruxolitinib in September 2020, capturing the period when researchers began investigating Ruxolitinib for managing severe COVID-19 symptoms.

Conclusion and Future Work

TextEssence demonstrates that stability is a prerequisite for meaningful qualitative discovery in embedding analysis. By combining interactive visualization with the $EC@k$ metric, the tool allows researchers to confidently track the evolution of discourse in rapidly changing fields. Future work will focus on automating the discovery of high-shift concepts and adapting the replicate-based confidence approach for more computationally intensive Transformer-based models.

Summary updated

Abstract

Embeddings of words and concepts capture syntactic and semantic regularities of language; however, they have seen limited use as tools to study characteristics of different corpora and how they relate to one another. We introduce TextEssence, an interactive system designed to enable comparative analysis of corpora using embeddings. TextEssence includes visual, neighbor-based, and similarity-based modes of embedding analysis in a lightweight, web-based interface. We further propose a new measure of embedding confidence based on nearest neighborhood overlap, to assist in identifying high-quality embeddings for corpus analysis. A case study on COVID-19 scientific literature illustrates the utility of the system. TextEssence is available from https://github.com/drgriffis/text-essence.
BibTeX
@inproceedings{newmangriffis2021textessence,
  author = {Denis Newman-Griffis and Venkatesh Sivaraman and Adam Perer and Eric Fosler-Lussier and Harry Hochheiser},
  title = {{TextEssence: A Tool for Interactive Analysis of Semantic Shifts Between Corpora}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-demos.13},
}
Figure