Modeling Document and Section Hierarchies in Embedding Space

Kriste Krstovski, Sharyn O'Halloran, Kshitij ChandnaarXiv 2026 · arXiv

TL;DR

The Hierarchical Embedded Topic Model (hETM) moves beyond treating documents as flat "bags of words" by explicitly modeling their internal section structure. By coupling section-level and document-level topic distributions through a hierarchical prior in a shared embedding space, hETM captures both sharp subtopics and overarching themes, outperforming standard models on complex, structured text like legal and financial reports.

What Problem Does This Paper Address?

Most topic models, from the classic Latent Dirichlet Allocation (LDA) to modern neural variants like the Embedded Topic Model (ETM), treat a document as a single, flat distribution over topics. This ignores the reality of many real-world document types—such as legal statutes, financial 10-K filings, or long forum threads—which are organized into sections that often address distinct subtopics.

Treating these documents as flat forces a difficult trade-off:

  1. Modeling the whole document blurs the specific topics of individual sections into a generic "average."
  2. Modeling each section independently loses the statistical strength and the "big picture" theme that ties the sections together.

The authors aim to solve this by creating a model that captures both levels of meaning simultaneously, particularly for documents with large, specialized vocabularies where traditional methods often fail.

Methods

The authors introduce hETM, which extends the Embedded Topic Model (ETM) into a hierarchical framework.

  • Embedding-Based Topics: Like ETM, hETM represents topics as points in a word embedding space. The probability of a word given a topic is determined by the inner product of their embeddings. This allows the model to handle "heavy-tailed" vocabularies (rare words) effectively.
  • Hierarchical Prior in Logit Space: Instead of a single topic distribution per document, hETM assigns a distribution to the document ($\theta_d$) and a separate distribution to each of its sections ($\theta_s$). Crucially, these are coupled: the "logit" (pre-softmax representation) of a section is drawn from a Gaussian distribution centered on the document's logit. This creates a "shrinkage" effect where section topics are encouraged to stay near the document's overall theme but are free to vary.
  • Neural Inference: The model uses amortized variational inference. It employs two neural network encoders—one for the document and one for sections—to predict the parameters of the topic distributions.
  • Closed-Form Coupling: A key technical contribution is deriving a variational evidence lower bound (ELBO) that includes a closed-form term for the level-coupling, making training efficient and stable.

Datasets

The model was evaluated on four diverse corpora representing different hierarchical structures:

  • U.S. Trade Legislation: 576 bills with ~40k sections. Highly structured with distinct legal subtopics.
  • Edgar (Financial Reports): 4,435 10-K filings. Long documents with standardized sections (e.g., "Risk Factors").
  • EUR-Lex (EU Legislation): 10,000 documents with ~75k sections.
  • 20 Newsgroups: A classic dataset used here to represent an implicit hierarchy, where a "document" is a newsgroup and "sections" are individual posts.

Key Insights and Ideas

  • Sharpness vs. Coherence: By tying section topics to a document-level prior, the section topics remain "sharp" (focused), while the document-level distribution emerges as a coherent, smoothed aggregate. This is clearly visible in the model's representation of complex acts.
Figure 1: hETM representation of the Omnibus Trade and Competitiveness Act of 1988 (50 topics): the docu- ment distribution θd and three example sections θs with the eight most probable words of their top topics.
  • Embeddings as a Bridge: Using a shared embedding space for both levels allows the model to share semantic information across the hierarchy. This is especially useful for specialized domains (legal, finance) where word co-occurrence alone isn't enough.
  • Reparameterization is Key: By placing the hierarchy in the logit space rather than using a nested Dirichlet prior (as in older models like mlhLDA), the authors keep the model fully reparameterizable, enabling the use of modern deep learning optimization techniques.

Key Results

hETM consistently outperformed both flat and hierarchical baselines across several metrics:

  • Perplexity: hETM achieved the lowest or near-lowest held-out perplexity on almost all datasets and topic counts ($K$), indicating a much better fit to the data than flat models like ETM or older hierarchical models like mlhLDA.
Figure 2: Held-out perplexity (section completion) as a function of the number of topics K for ETM(D), ETM(S), hETM, SawETM, TSNTM, and mlhLDA on the four document corpora.
  • Topic Quality: The model produced topics with high coherence and diversity. Qualitatively, hETM recovered clean, distinctive themes (e.g., "Agriculture," "Banking") while baselines often mixed in procedural "boilerplate" text.
Topic quality (coherence x diversity) across datasets.
  • Downstream Clustering: In a document clustering task, hETM achieved the highest Adjusted Rand Index (ARI) on the U.S. Trade and Edgar datasets, proving that its hierarchical representations are more semantically meaningful for downstream applications.

Limitations and Future Work

  • Fixed Partitions: hETM assumes documents are already split into sections. If the partition is implicit, it relies on heuristic construction.
  • Bag-of-Words Limitation: Like most topic models, it ignores word order within sections, missing out on local syntactic context.
  • Two-Level Hierarchy: The current work focuses on a two-level (document-section) hierarchy. Extending this to deeper hierarchies (e.g., Chapter -> Section -> Paragraph) remains a direction for future research.
  • Section Order: The model treats sections as conditionally independent, ignoring the sequential flow of a document. The authors suggest that incorporating a sequential prior over section logits is a natural next step.

Summary updated

Abstract

Embedded topic models integrate word embeddings into the topic generation process, producing coherent topics and robust performance on large vocabularies. However, each document is represented by a single topic distribution, ignoring its internal structure. Many real world collections, including legal documents, company and government reports, and online discussion forums, exhibit hierarchical organization in which documents consist of multiple sections that often address distinct subtopics. We introduce the Hierarchical Embedded Topic Model (hETM), a structured topic model that couples the topic distribution of a document and those of its sections through a hierarchical prior in the embedding space. We derive an amortized variational inference procedure for hETM and evaluate it on structured collections from each of these domains. Experimental results show that hETM achieves the lowest or near-lowest held-out perplexity among flat and hierarchical topic model baselines, along with comparable or higher topic quality. It also performs strongly on a downstream clustering task, achieving the best clustering performance on most of the collections.
BibTeX
@misc{krstovski2026modeling,
  author = {Kriste Krstovski and Sharyn O'Halloran and Kshitij Chandna},
  title = {{Modeling Document and Section Hierarchies in Embedding Space}},
  year = {2026},
  eprint = {2611.21980},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21980},
}
Figure