Introduction
Word embeddings are a cornerstone of modern Natural Language Processing (NLP), providing mathematical representations of words based on their linguistic context. However, using these embeddings to compare different text collections—such as scientific literature from different time periods—presents significant challenges. Most existing methods focus on quantitative metrics that indicate that a change occurred, but fail to provide qualitative insights into how meanings shifted. Furthermore, embedding models are notoriously unstable; training the same model twice on the same data can yield different results, making it difficult to distinguish genuine semantic shifts from algorithmic noise.
To address these issues, we present TextEssence, an interactive web-based tool designed for the visual and qualitative exploration of semantic shifts across corpora. TextEssence introduces a novel "embedding confidence" metric to filter out unstable results, ensuring that the shifts observed by researchers are statistically robust.
The TextEssence Interactive Workflow
TextEssence is built with a Python backend and a Svelte frontend, offering three primary modes of exploration to help researchers navigate complex datasets:
- Browse: Users can view a 2D map of concepts generated via t-SNE. To ensure meaningful comparisons, the tool uses a Procrustes transformation to align maps from different corpora. Visual "trajectories" show how concepts move between datasets, with line thickness indicating the magnitude of the shift.
- Inspect: This mode allows for a deep dive into the "neighborhood" of a specific concept—the words most similar to it—to see how its immediate context evolves over time.
- Compare: Researchers can track the relationship between two specific concepts across multiple datasets, identifying when and how their similarity peaks or declines.
Technical Innovation: Embedding Confidence ($EC@k$)
A key contribution of TextEssence is the Embedding Confidence ($EC@k$) metric. To mitigate the instability of embedding algorithms, the system trains multiple "replicates" (e.g., 10 versions) of the same model. It then measures the overlap of a word's nearest neighbors across these replicates. If a word's neighbors are consistent across all runs, it is assigned high confidence; if they fluctuate, the result is considered unreliable and filtered from the visualization.
By requiring a confidence threshold (typically 0.5), TextEssence ensures that users only analyze stable, reproducible semantic patterns. While modern contextualized models like BERT are popular, TextEssence utilizes static embeddings (e.g., word2vec-style) because they are more resource-efficient for training the necessary replicates and better at capturing aggregate corpus-level patterns.
Case Study: COVID-19 Scientific Literature
We applied TextEssence to the CORD-19 (COVID-19 Open Research Dataset), analyzing monthly snapshots of scientific papers from March to October 2020. To improve conceptual clarity, we used ScispaCy and JET to map text to SNOMED CT medical codes, allowing the tool to treat synonymous terms (e.g., "Anosmia" and "Loss of smell") as single entities.
The analysis revealed several key scientific shifts:
- Anosmia: Early in the pandemic, its neighbors were general symptoms like "taste" and "pharyngeal pain." By October, it shifted toward systemic COVID-19 symptoms like "nausea," reflecting its emergence as a hallmark diagnostic indicator.
- Dexamethasone: Initially associated with "traumatic brain injury," this steroid moved into a stable neighborhood of respiratory treatments and antivirals following clinical trials that proved its efficacy for COVID-19.
- Ruxolitinib: The tool identified a sharp increase in similarity between Dexamethasone and Ruxolitinib in September 2020, capturing the period when researchers began investigating Ruxolitinib for managing severe COVID-19 symptoms.
Conclusion and Future Work
TextEssence demonstrates that stability is a prerequisite for meaningful qualitative discovery in embedding analysis. By combining interactive visualization with the $EC@k$ metric, the tool allows researchers to confidently track the evolution of discourse in rapidly changing fields. Future work will focus on automating the discovery of high-shift concepts and adapting the replicate-based confidence approach for more computationally intensive Transformer-based models.
Summary updated