Abstract
Dimension reduction (DR) can transform high-dimensional text embeddings into a 2D visual projection facilitating the exploration of document similarities. However, the projection often lacks connection to the text semantics, due to the opaque nature of text embeddings and non-linear dimension reductions. To address these problems, we propose a gradient-based method for visualizing the spatial semantics of dimensionally reduced text embeddings. This method employs gradients to assess the sensitivity of the projected documents with respect to the underlying words. The method can be applied to existing DR algorithms and text embedding models. Using these gradients, we designed a visualization system that incorporates spatial word clouds into the document projection space to illustrate the impactful text features. We further present three usage scenarios that demonstrate the practical applications of our system to facilitate the discovery and interpretation of underlying semantics in text projections.
Introduction
Dimension Reduction (DR) techniques like t-SNE and MDS are staples in data science for visualizing high-dimensional data. In Natural Language Processing (NLP), these techniques map complex, high-dimensional text embeddings (e.g., from BERT) into 2D "star maps" to reveal document similarities. However, these visualizations are often criticized for being "black boxes." While they show that documents are similar by placing them close together, they fail to explain why they are positioned that way. This interpretability gap stems from two sources: the abstract nature of modern embeddings and the non-linear complexity of DR algorithms. This paper introduces a gradient-based approach to bridge this gap, providing a way to see the "spatial semantics"—the actual meaning—of the 2D layout.
Gradient-Based Methodology
The core innovation of this work is the use of backward-mode automatic differentiation (autodiff) to calculate "projection gradients." By treating the entire pipeline—from input words to embedding to 2D projection—as a single differentiable function, we can determine which specific words "pull" a document toward its position on the map.
As shown in the pipeline above, the system calculates the partial derivative of the 2D coordinates $(v_x, v_y)$ with respect to each input word. This provides a vector for every word, indicating the magnitude and direction of its influence on the document's final placement. Leveraging modern autodiff libraries like PyTorch makes this process highly efficient, requiring only two passes (one for each dimension) regardless of document length.
Visualizing Semantics
Using these gradients, the system generates two primary types of visualizations:
- Spatial Word Clouds: These aggregate the most impactful words across the projection space, placing them at the centroids of the documents they influence to label clusters semantically.
- Impact Heatmaps: These highlight specific words within a document based on their gradient magnitude, showing which terms were most critical for that document's specific location.
Evaluation and Key Insights
The system was evaluated using the COVID-19 Open Research Articles and BBC News datasets. A key finding was the distinction between gradient-based explanations and standard attention weights. While attention weights reflect what the model deems important, gradients reflect what is important for the final 2D layout. Consequently, gradient-based word clouds adapt to the specific DR algorithm used (e.g., MDS vs. t-SNE), whereas attention-based clouds remain static.
In the COVID-19 case study, the system successfully identified clear clusters around terms like "smoking" and "neurological" in fine-tuned models, which were previously indistinguishable. Performance testing showed that enabling autograd for these calculations adds only approximately 12% overhead to execution time, making it a viable tool for real-time exploratory analysis.
Conclusion and Future Work
This gradient-based approach provides a scalable and efficient method for interpreting opaque text projections. While the current system effectively labels clusters, future work will address visual overlap in dense regions through advanced label-placement strategies and explore "semantic zooming" to allow users to interactively discover sub-topics within the projection space.
Summary updated