SAP-sLDA: An Interpretable Interface for Exploring Unstructured Text

Charumathi Badrinath, Weiwei Pan, Finale Doshi-VelezarXiv 2023 · DOI · arXiv

Introduction

Exploring massive collections of unstructured text, such as the "Dharma Seed" repository of Buddhist teachings, typically involves a pipeline of Latent Dirichlet Allocation (LDA) followed by dimensionality reduction techniques like t-SNE or PCA. However, this standard approach suffers from two critical issues: semantic mismatch and instability. The clusters generated often fail to align with human intuition, and the non-identifiability of LDA means that different runs on the same data produce wildly different maps, creating an unreliable and confusing interface for users.

Standard LDA projections across multiple runs, illustrating the instability and lack of clear semantic clustering.

Methodology: Semantically-Aligned-Projection sLDA

To address these flaws, the authors propose SAP-sLDA (Semantically-Aligned-Projection supervised LDA). This semi-supervised approach incorporates a human-in-the-loop (HiTL) feedback cycle to regularize the model and ensure the final visualization is both stable and meaningful.

The Regularization Term

Unlike standard LDA, which optimizes solely for how well topics explain words, SAP-sLDA introduces a spatial interpretability term to the objective function. This term penalizes the model when documents with identical semantic labels are projected far apart in 2D space, or when documents with different labels are clustered too closely. This specifically optimizes the spatial arrangement of the final 2D projection that the user actually sees.

Active Learning via Variance-Based Labeling

To minimize human effort, the authors utilize a variance-based labeling strategy. By identifying documents whose positions in the 2D projection shift most significantly across random restarts—essentially finding the documents the model is most "confused" about—the system targets them for human labeling. This strategy proved highly efficient: on synthetic data, labeling just 15% of documents using this method outperformed labeling 25% of documents at random.

Key Results and Insights

Experiments on both synthetic data and the Dharma Seed corpus demonstrate that SAP-sLDA successfully recovers ground-truth themes where standard LDA fails.

Improved Stability and Purity

A core finding is that anchoring even a small number of points with human labels dramatically increases the robustness of the entire map across different runs. Qualitatively, the resulting clusters are significantly "purer." For instance, in the Dharma Seed corpus, documents related to specific themes like "Practices of Mindfulness" remained tightly grouped in SAP-sLDA projections, whereas they were scattered in baseline LDA results.

Comparison of topic distributions for LDA, pf-sLDA, and SAP-sLDA, showing the increased purity and alignment of the proposed method.

Comparison with Baselines

As shown in the topic distribution heatmaps, SAP-sLDA produces much more distinct and interpretable topic assignments compared to standard LDA and prediction-focused supervised LDA (pf-sLDA). While pf-sLDA improves upon standard LDA, it often results in artificially sparse distributions and remains less stable in complex, real-world scenarios.

Conclusion

SAP-sLDA offers a promising path toward more reliable and intuitive text exploration tools. By treating stability as a primary feature and leveraging smart, sparse human feedback, it bridges the gap between mathematical topic modeling and human semantic understanding. Future work will focus on systematic hyperparameter tuning and formal user studies to evaluate the interface's utility for community-driven library navigation.

Summary updated

Abstract

A common way to explore text corpora is through low-dimensional projections of the documents, where one hopes that thematically similar documents will be clustered together in the projected space. However, popular algorithms for dimensionality reduction of text corpora, like Latent Dirichlet Allocation (LDA), often produce projections that do not capture human notions of document similarity. We propose a semi-supervised human-in-the-loop LDA-based method for learning topics that preserve semantically meaningful relationships between documents in low-dimensional projections. On synthetic corpora, our method yields more interpretable projections than baseline methods with only a fraction of labels provided. On a real corpus, we obtain qualitatively similar results.
BibTeX
@misc{badrinath2023sap,
  author = {Charumathi Badrinath and Weiwei Pan and Finale Doshi-Velez},
  title = {{SAP-sLDA: An Interpretable Interface for Exploring Unstructured Text}},
  year = {2023},
  eprint = {2308.01420},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.01420},
}
Figure