Abstract
This paper introduces a novel end-to-end pipeline for the multi-scale semantic organization of large text collections. By leveraging the high-dimensional representational power of Large Language Model (LLM) embeddings and an iterative density-based clustering algorithm, the proposed method constructs a hierarchical "semantic tree." This approach avoids the limitations of rigid, pre-defined taxonomies by allowing thematic structures to emerge naturally from the data density. We demonstrate the effectiveness of this method across diverse datasets, showing that it successfully captures both fine-grained concepts and broad thematic regions while providing human-readable annotations via automated LLM labeling.
Introduction
As digital libraries and research repositories expand, the challenge of organizing information effectively becomes a "Goldilocks" problem. Traditional flat clustering methods, such as k-means, often provide a single level of resolution that is either too specific or too broad. Conversely, fixed human-curated taxonomies are often rigid and struggle to adapt to emerging interdisciplinary fields. This research addresses the need for multi-scale semantic organization, seeking a method that can automatically discover how specific concepts (e.g., "quantum optics") aggregate into broader themes (e.g., "Physics") and eventually into global domains.
Methodology
The proposed pipeline consists of four primary stages designed to transform raw text into an interpretable hierarchy.
1. Embedding and Dimensionality Reduction
Documents are first converted into high-dimensional vectors using state-of-the-art LLM embedding models, such as Qwen3-Embedding-8B or SFR-Embedding-Mistral. To improve the clarity of the resulting structure, the authors apply Principal Component Analysis (PCA). Contrary to the intuition that raw high-dimensional cosine distance is superior, reducing the data to a few "most informative" dimensions (typically 2 to 10) acts as a crucial feature-selection mechanism, filtering out semantic noise.
2. Adaptive Density Clustering
The core of the hierarchical inference is an iterative application of DBSCAN. The process begins with a strict (small) radius, identifying very dense clusters of highly similar documents. As the radius ($\epsilon$) is progressively increased, these tight clusters merge into more diffuse groups, naturally forming a dendrogram or tree structure.
3. Automated Annotation
To ensure the tree is human-interpretable, each node is labeled using a large-context LLM. The system employs a decision logic based on the number of abstracts and the quality of the generated label.
As shown in the workflow, if a cluster contains fewer than 1500 abstracts, they are processed directly by Llama4Scout17B16E-Instruct. For larger clusters, a sampling strategy is used. The resulting labels are further refined or summarized using gpt-oss-20b if they do not meet specific length constraints (e.g., < 1 line and < 80 characters), ensuring concise and consistent node descriptions.
Experimental Evaluation
The method was evaluated on a diverse set of corpora, including institutional research abstracts from the American University of Beirut (AUB) and TU Wien, as well as standard benchmarks like 20 Newsgroups, AG News, and IMDB 50K.
Key Insights
- Optimal Resolution: Semantic agreement with ground-truth labels (measured by ARI and NMI) typically peaks at intermediate density levels. This suggests that human-defined categories correspond to a specific "natural" resolution in the embedding space.
- Topic vs. Sentiment: Experiments on the IMDB dataset revealed that current LLM embeddings prioritize topical content over sentiment. The hierarchy effectively separated movie genres but failed to distinguish between positive and negative reviews at a structural level.
- Taxonomy Alignment: On the Theses-fr-en dataset, the unsupervised tree showed strong alignment with the OECD "Fields of Science" classification, demonstrating its ability to recover professional academic structures without prior knowledge.
Conclusion
This research demonstrates that combining nested density clustering with LLM-based embeddings and annotation provides a powerful, scalable tool for mapping complex semantic landscapes. While the computational cost of LLM annotation remains a factor, the ability to discover emergent interdisciplinary connections offers significant advantages over traditional static taxonomies. Future work will explore extending this density-relaxation approach to non-textual modalities like image and audio embeddings.
Summary updated