TL;DR
The authors introduce the Nested Hierarchical Dirichlet Process (nHDP), a flexible framework for hierarchical topic modeling. Unlike previous models that force an entire document to follow a single path down a topic tree, the nHDP allows individual words to traverse their own paths, enabling documents to "borrow" themes from different branches of a shared global hierarchy.
What Problem Does This Paper Address?
In traditional hierarchical topic modeling, the nested Chinese Restaurant Process (nCRP) is the gold standard. However, the nCRP has a significant limitation: it assumes that all topics within a single document must lie along a single path from the root of a tree to a leaf.
Real-world data is rarely that tidy. A news article about a stadium's construction might primarily belong to a "Sports" hierarchy, but it will inevitably "borrow" terms from "Architecture," "Finance," or "Local Government." The nCRP struggles with these cross-thematic borrowings because it performs document-specific path clustering.
The authors aim to move toward word-specific path clustering, allowing for more nuanced and realistic representations of complex documents. As illustrated in the figure above, while the traditional nCRP restricts a document to a single path (top row), the proposed nHDP allows a document's words to populate multiple branches of the topic tree (bottom row).
Methods
The paper proposes the nHDP, which generalizes the nCRP by incorporating the Hierarchical Dirichlet Process (HDP).
- Global Tree: The model starts with a global tree structure (an nCRP) that represents the universal hierarchy of topics.
- Document-Specific Trees: Instead of picking one path, each document is assigned its own distribution over the entire global tree. This is achieved by treating the global nCRP as a base distribution for a second level of Dirichlet processes.
- Word-Level Random Walks: For every word in a document, the model performs a random walk starting at the root. At each node, a document-specific switching probability (modeled as a Beta random variable) determines whether the word stops at the current topic or continues deeper into a more specific sub-topic.
- Inference: To handle the massive scale of their dataset, the authors utilize a scalable variational inference algorithm, allowing the model to process millions of documents efficiently.
Datasets
The model was evaluated on a massive corpus of 1.8 million documents from The New York Times. This dataset is a standard benchmark for large-scale topic modeling due to its wide variety of professional, human-labeled, and hierarchically organized content.
Key Insights and Ideas
- From Documents to Words: The core "aha moment" is shifting the unit of hierarchical assignment from the document level to the word level. This allows a document to have a "dominant" path while still allowing "offshoot" words to inhabit other relevant branches.
- Thematic Borrowing: By allowing words to wander, the model captures "random effects"—the specific, sometimes unexpected vocabulary that appears when different high-level themes intersect.
- Hierarchical Specificity: The model naturally enforces a "general-to-specific" structure. Topics near the root contain general terms (e.g., "government," "officials"), while leaf nodes contain highly specific entities (e.g., "Yankees," "Mets," "Jeter").
Key Results
The authors provide qualitative evidence of the model's success through a visualization of the learned hierarchy:
- Meaningful Branching: The model successfully clustered topics into logical subtrees. For instance, a "Sports" subtree branched into specific sports (Baseball, Hockey), which then branched further into specific teams and players.
- Regional Logic: In the "Foreign Affairs" subtree, the model grouped topics by major subregions (e.g., Middle East, Europe) and then branched into specific regional issues or conflicts.
- Scalability: The authors demonstrated that Bayesian nonparametric models, often criticized for being computationally expensive, can be scaled to nearly 2 million documents using modern inference techniques.
Limitations and Future Work
- Workshop Scope: As this is a workshop version of a longer paper, the technical details of the scalable inference algorithm and quantitative comparisons (like perplexity scores) are abbreviated and pointed toward a longer reference paper.
- Tree Topology: The model assumes an infinite branching factor at each node, but the depth of the tree or the "optimal" switching probabilities for different types of corpora remains an area for further tuning.
- Future Directions: The authors suggest that the scalable inference framework used here is the key to making these complex hierarchical models practical for industry-scale datasets.
Summary updated