TL;DR
The Hierarchical Embedded Topic Model (hETM) moves beyond treating documents as flat "bags of words" by explicitly modeling their internal section structure. By coupling section-level and document-level topic distributions through a hierarchical prior in a shared embedding space, hETM captures both sharp subtopics and overarching themes, outperforming standard models on complex, structured text like legal and financial reports.
What Problem Does This Paper Address?
Most topic models, from the classic Latent Dirichlet Allocation (LDA) to modern neural variants like the Embedded Topic Model (ETM), treat a document as a single, flat distribution over topics. This ignores the reality of many real-world document types—such as legal statutes, financial 10-K filings, or long forum threads—which are organized into sections that often address distinct subtopics.
Treating these documents as flat forces a difficult trade-off:
- Modeling the whole document blurs the specific topics of individual sections into a generic "average."
- Modeling each section independently loses the statistical strength and the "big picture" theme that ties the sections together.
The authors aim to solve this by creating a model that captures both levels of meaning simultaneously, particularly for documents with large, specialized vocabularies where traditional methods often fail.
Methods
The authors introduce hETM, which extends the Embedded Topic Model (ETM) into a hierarchical framework.
- Embedding-Based Topics: Like ETM, hETM represents topics as points in a word embedding space. The probability of a word given a topic is determined by the inner product of their embeddings. This allows the model to handle "heavy-tailed" vocabularies (rare words) effectively.
- Hierarchical Prior in Logit Space: Instead of a single topic distribution per document, hETM assigns a distribution to the document ($\theta_d$) and a separate distribution to each of its sections ($\theta_s$). Crucially, these are coupled: the "logit" (pre-softmax representation) of a section is drawn from a Gaussian distribution centered on the document's logit. This creates a "shrinkage" effect where section topics are encouraged to stay near the document's overall theme but are free to vary.
- Neural Inference: The model uses amortized variational inference. It employs two neural network encoders—one for the document and one for sections—to predict the parameters of the topic distributions.
- Closed-Form Coupling: A key technical contribution is deriving a variational evidence lower bound (ELBO) that includes a closed-form term for the level-coupling, making training efficient and stable.
Datasets
The model was evaluated on four diverse corpora representing different hierarchical structures:
- U.S. Trade Legislation: 576 bills with ~40k sections. Highly structured with distinct legal subtopics.
- Edgar (Financial Reports): 4,435 10-K filings. Long documents with standardized sections (e.g., "Risk Factors").
- EUR-Lex (EU Legislation): 10,000 documents with ~75k sections.
- 20 Newsgroups: A classic dataset used here to represent an implicit hierarchy, where a "document" is a newsgroup and "sections" are individual posts.
Key Insights and Ideas
- Sharpness vs. Coherence: By tying section topics to a document-level prior, the section topics remain "sharp" (focused), while the document-level distribution emerges as a coherent, smoothed aggregate. This is clearly visible in the model's representation of complex acts.
- Embeddings as a Bridge: Using a shared embedding space for both levels allows the model to share semantic information across the hierarchy. This is especially useful for specialized domains (legal, finance) where word co-occurrence alone isn't enough.
- Reparameterization is Key: By placing the hierarchy in the logit space rather than using a nested Dirichlet prior (as in older models like mlhLDA), the authors keep the model fully reparameterizable, enabling the use of modern deep learning optimization techniques.
Key Results
hETM consistently outperformed both flat and hierarchical baselines across several metrics:
- Perplexity: hETM achieved the lowest or near-lowest held-out perplexity on almost all datasets and topic counts ($K$), indicating a much better fit to the data than flat models like ETM or older hierarchical models like mlhLDA.
- Topic Quality: The model produced topics with high coherence and diversity. Qualitatively, hETM recovered clean, distinctive themes (e.g., "Agriculture," "Banking") while baselines often mixed in procedural "boilerplate" text.
- Downstream Clustering: In a document clustering task, hETM achieved the highest Adjusted Rand Index (ARI) on the U.S. Trade and Edgar datasets, proving that its hierarchical representations are more semantically meaningful for downstream applications.
Limitations and Future Work
- Fixed Partitions: hETM assumes documents are already split into sections. If the partition is implicit, it relies on heuristic construction.
- Bag-of-Words Limitation: Like most topic models, it ignores word order within sections, missing out on local syntactic context.
- Two-Level Hierarchy: The current work focuses on a two-level (document-section) hierarchy. Extending this to deeper hierarchies (e.g., Chapter -> Section -> Paragraph) remains a direction for future research.
- Section Order: The model treats sections as conditionally independent, ignoring the sequential flow of a document. The authors suggest that incorporating a sequential prior over section logits is a natural next step.
Summary updated