Hierarchical Topic Modeling for Nested Text Representations

102 concepts95 papers1 proposal

When we organize human knowledge, we rarely use a flat, unordered list. Instead, we naturally construct hierarchies: a broad domain like Science branches into Biology, which further narrows into Genetics.

Traditional topic modeling techniques, such as standard Latent Dirichlet Allocation (LDA), ignore this structure. They treat topics as independent, co-equal buckets of words. When forced to model a complex corpus, a flat topic model faces a difficult dilemma: it must either lump general and specific concepts into the same vague category, or fragment a single coherent theme into redundant, disconnected sub-topics.

To capture the true multi-layered nature of text, researchers developed Hierarchical Topic Modeling. This paradigm shifts the goal from discovering what topics exist to discovering how they nest within one another.

The Intuition: Generative Paths and Level Selection

To understand why hierarchical topic models work, we must look at how they change the story of how a document is generated. In a flat topic model, a document is a mixture of topics, and each word is drawn directly from one of those topics.

In a hierarchical model, the topics are arranged as nodes in a tree. The root node contains the most general language common to the entire corpus (words like system, data, or approach). As you move down the branches, the nodes become progressively more specific.

The seminal framework by Blei et al. introduced this using a non-parametric Bayesian prior called the nested Chinese Restaurant Process (nCRP). The intuition works as follows:

  1. Path Assignment: Each document is assigned a single path from the root of the tree down to a leaf node. This path represents the document's specific thematic lineage (e.g., Technology $\rightarrow$ Open Source $\rightarrow$ Linux).
  2. Level Selection: When generating a word, the document does not choose from all possible topics. Instead, it flips a coin (or samples from a Dirichlet distribution) to decide which level of its assigned path to draw from.
  3. Word Generation: If it selects the top level, it draws a word from the general Technology distribution. If it selects the bottom level, it draws from the highly specialized Linux distribution.

By separating words based on their generality, the model naturally filters out background vocabulary at the top and isolates distinct, granular concepts at the bottom. This allows unsupervised models to discover both the content of the topics and the structure of the tree itself.

Scaling Up: The Neural Transition and Posterior Collapse

While early Bayesian models provided a elegant mathematical foundation, they relied on computationally expensive inference techniques like Gibbs sampling, making them difficult to scale to massive modern datasets. To solve this, researchers transitioned to Neural Topic Models (NTMs), which leverage Variational Autoencoders (VAEs) to learn topic distributions via neural networks.

However, stacking layers in a deep neural network to create a topic hierarchy introduces a major technical hurdle: posterior collapse. As highlighted by Li et al., when a variational model is trained, the latent variables in the deeper layers often become inactive or redundant. The network learns to reconstruct the text using only the top layers, causing the hierarchy to collapse into a flat, non-varying representation.

To prevent this collapse and enforce a strict progression of granularity, modern architectures build structural constraints directly into the network's logic:

  • Tree-Structured Networks: The Tree-Structured Neural Topic Model (TSNTM) proposed by Mori et al., along with its nonparametric variant developed by Chen et al., uses a stick-breaking process to dynamically determine the branching factor at each layer, forcing child nodes to specialize in distinct sub-compartments of their parent's semantic space.
  • Layered Belief Networks: Frameworks like the Dirichlet Belief Network (DirBN) developed by Zhao et al. construct deep generative paths where each layer explicitly models a different level of word abstraction, ensuring that deeper layers cannot simply replicate the broad distributions of the upper layers.

Geometric and Embedding-Based Hierarchies

Parallel to probabilistic modeling, the rise of dense text embeddings opened a new frontier: capturing hierarchy geometrically. Instead of relying purely on word-co-occurrence statistics, these models arrange continuous vector spaces so that distance and direction reflect semantic depth.

For instance, Meng et al. demonstrated that by using spherical text embeddings, a model can measure "word specificity." In this geometric setup, broad parent concepts sit at the center of a cluster, while highly specific terms push outward toward the periphery. This allows discriminative topic mining using only a few high-level category names as guidance.

To optimize these multi-level spaces without relying on massive, slow language models, Vuong introduced a framework called HiCOT. It combines Optimal Transport—a mathematical method for calculating the minimal cost of transforming one probability distribution into another—with contrastive learning. By minimizing the transport distance between related parent and child topics across layers, the model ensures smooth semantic transitions while maintaining sharp distinctions between different levels of abstraction.

Guiding Trees with External Knowledge

While discovering hierarchies completely from scratch is powerful, the resulting trees can sometimes be unpredictable or counterintuitive to human users. To bridge this gap, a significant branch of research focuses on guiding the hierarchical process using pre-existing human taxonomies.

Instead of learning a tree blindly, models can ingest the curated structures of web directories (Petinot et al.) or the Wikipedia Category Network (Hu et al.; Kumar et al.). These hybrid approaches use the human-curated network as a structural scaffolding, while the underlying topic model fills in the granular, data-driven word distributions. This combines the interpretability of a manual taxonomy with the coverage and adaptability of an automated statistical model.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 96 papers
  2. Version 1 · 2026-10-04 · 74 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_topic_modeling_7355c859,
  title = {hierarchical topic modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-topic-modeling/rdr_7355c8595e284b528b72a4a9213bc219},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_hierarchical_topic_modeling_7355c859_v2,
  title = {hierarchical topic modeling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-topic-modeling/rdr_7355c8595e284b528b72a4a9213bc219/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake