Testing Topic Labels Against the Document Graph

Topic models group text into themes, and large language models now write the names for those themes. The names are fluent whether or not the underlying group makes sense. A cluster that mixes two unrelated themes, or that just collects passages from one novel, still comes back with a tidy phrase like "Maritime Hardship and Family Loss." The standard automatic quality scores do not catch this. They count how often a topic's top words appear together, and humans' ability to label a topic barely tracks them. This note proposes treating each generated label as a claim that can be checked. A label is good if a reader holding only the label can pick the topic's passages out from a set of decoys. The decoys come from the documents' own structure: other passages from the same section, the same author, or the same hashtag thread that belong to a different topic.

The idea in two sentences: build a graph linking sentences to their paragraphs, sections, documents, entities, and authors. Then score every topic label by how well it separates true members from structurally adjacent non-members. A label that only works against random decoys is naming the document rather than the theme, and a label that fails even against random decoys is naming nothing. The resulting score is meant to replace word-co-occurrence coherence when choosing topic models, flagging nonsense topics, and merging near-duplicate labels.

Core Claim

Our central claim is that a topic label's quality is a behavioral property, namely how well it predicts membership. The hardest and most informative test of that property uses negatives drawn from the heterogeneous document graph. A heterogeneous document graph is a graph whose nodes are different kinds of things (words, sentences, sections, documents, entities, users) connected by typed edges.

We make three specific predictions. First, a "grounded labelability" score computed this way will correlate with human labeling agreement much better than normalized pointwise mutual information ($C_{NPMI}$) or $C_V$. Second, splitting the score by negative type will diagnose why a topic fails. Third, two labels that select the same members are the same topic regardless of wording, which gives a principled way to collapse the long tail of synonymous labels.

The motivation comes from three sources. Topic Model or Topic Twaddle? audited coherence metrics on 1.8M Australian political tweets (Auspol-18) against expert open labeling. It found that the metrics disagree on the best number of topics ($C_{NPMI}$ peaks at K=60 while $C_V$ keeps rising through K=200) and that their rank correlation with human agreement is low or zero. That study stops at diagnosis and builds no replacement.

FIDELITY clusters KeyBERT phrases and prompts LLaMA-2 70B with the top 50 keywords to name each cluster. It reports that the labeler hallucinates content absent from the keywords, and it has no mechanism to detect when this happens. Evaluating Large Language Models for Narrative Topic Labeling labels isolated 500-word passages zero-shot. It produces thousands of near-synonymous labels and needs a second GPT-4o pass to consolidate them. All three treat the label as the end product. We treat it as a hypothesis to test.

The testing idea is borrowed from interpretability work on neural features. There, Paulo et al. scale LLM explanation of model features, and Movva et al. show that contrastive exemplars (text where a feature fires versus closely matched text where it does not) let an LLM state testable hypotheses. We carry that "explain, then predict" logic over to topic models. Our addition is that the matched contrasts are chosen by document structure rather than by activation.

Research Program

The graph is built per corpus. For long documents, we parse a tree from document to section or chapter to paragraph to sentence, following the layout trees of DocNavRAG. DocNavRAG builds these trees to route questions, not to discover themes.

We add lateral sentence-to-entity and sentence-to-phrase edges. These follow the word-sentence bipartite design of HETSUMGRAPH, restricted to named entities and corpus-rare phrases. That paper notes that frequent words create spurious shortcuts between unrelated sentences, which is why we restrict the edges. For short text such as Auspol-18 there are no sections, so we use tweet, user, hashtag, and mention nodes; the dataset's own variants with and without hashtags and mentions make this natural. The graph is never trained. It serves only to sample decoys.

Any topic model can sit upstream. We will run LDA, BERTopic, SenClu (bag-of-sentences clustering; Efficient and Flexible Topic Modeling Using Pretrained Embeddings and Bag of Sentences), and FIDELITY unchanged. Each topic is labeled by an LLM from a sample of member sentences and contrastive non-members.

We then hold out a disjoint set of members and draw negatives from four strata. The first stratum is random corpus sentences. The second is embedding-nearest non-members, chosen without any reference to the graph. The third is graph-local non-members: same section, document, or author, but a different topic. The fourth is graph-bridged non-members: sentences sharing an entity or rare phrase with a positive. We also split positives into graph-near and graph-far relative to the labeling exemplars, so we can see whether the label generalizes across documents.

A judge model receives only the label and must rank candidates. We use a fixed mid-sized open LLM, plus a cheap cross-encoder variant to keep cost down. For a topic $t$ and stratum $s$, the score $A_s(t)$ is the area under the judge's ROC curve for positives versus stratum-$s$ negatives. The headline score is the worst-case stratum, $G(t)=\min_s A_s(t)$. We use the minimum because a label is only as trustworthy as its weakest discrimination. The individual $A_s$ values are kept as a diagnostic profile. High $A_{\text{random}}$ with low $A_{\text{local}}$ signals a topic that encodes document or author identity. Low $A_{\text{random}}$ signals twaddle.

For label consolidation, two labels are merged when the sets of held-out nodes their judges accept overlap heavily. The merged topic keeps whichever label scores higher on $G$. This comparison of what labels select, rather than how they are worded, replaces the second-pass LLM aggregation in the narrative-labeling work. Finally, a revision loop shows the labeler its own false positives from the weakest stratum and asks for a sharper label. It stops when $G$ stops improving, which directly targets FIDELITY's hallucination problem.

Baselines

The naive reference point is reporting $C_{NPMI}$ and $C_V$ on each topic's top words. These are what nearly every topic-modeling paper uses. The Twaddle study reports their Spearman correlation with human labeling agreement as low or statistically insignificant, falling to zero at small K. It does not transcribe exact values, so there is no reported numeric result on Lacuna for coherence-versus-$Q_{agr}$ correlation; we will recompute it on the same topics we score.

The strongest metric competitor is LLM-as-judge rating. The same judge model is given the top words and exemplars and asked to rate interpretability from 1 to 5, with an equal token budget and the same prompts tuned on the same development split. This competitor is strong because it uses the same model capacity as our method, minus the prediction test.

The strongest labeling pipeline is FIDELITY. It reports topic diversity (WE-PD) of 0.862 and inter-topic cosine similarity of 0.0686 on 20 Newsgroups, against 0.707 and 0.1981 for BERTopic. SenClu reports PMI 0.79 on 20 Newsgroups, against 0.35 for LDA and 0.2 for BERTopic. For narrative labeling, the best reported figure is GPT-4o's average human-preference rank of 2.40 on news (5 options, lower is better) in the narrative labeling study.

To beat these on the labeling side, revised labels must win head-to-head human preference against FIDELITY labels for the same clusters. They must do so without lowering WE-PD below FIDELITY's reported 0.862 on 20 Newsgroups, which shows the gain does not come from collapsing topics.

The key ablation replaces graph-sampled negatives with embedding-nearest negatives at an equal count. Everything else stays fixed: judge model, number of candidates, and labeling exemplars. This isolates the graph's contribution.

The proposal beats the metric baselines if $G$ reaches a significantly higher Spearman correlation with human agreement than both coherence and LLM-as-judge rating. Significance is assessed by a paired bootstrap over topics on the same models and K values.

Why not just…?

Why not just ask an LLM to rate how interpretable each topic is? This is the most serious threat. LLM judges are cheap and might already track human judgment. The failure mode we expect is the one FIDELITY observed in its labeler: a model rates its own fluent story about noisy words as sensible. A prediction test removes the opportunity to be persuaded by fluency. The head-to-head correlation experiment on Auspol-18 settles the question directly. If rating matches $G$, we will report that and recommend the cheaper method.

Why not just use hard negatives from embedding similarity, with no graph at all? This could plausibly work almost as well, because nearest neighbors in embedding space often share a document. If the ablation shows $A_{\text{embed}}$ and $A_{\text{local}}$ are interchangeable, the graph is unnecessary and the contribution shrinks to "predictive label scoring with hard negatives." We make one falsifiable prediction about where they differ. On chaptered fiction and on author-heavy tweets, topics that encode document or author identity will pass embedding-nearest negatives but fail same-document, different-topic negatives. We test this with planted identity topics in the synthetic experiment.

Why not just put the graph structure into the labeling prompt? This is the obvious combination of the three directions, giving the labeler section paths and neighboring sentences. We include it as an extra labeler variant. However, it does not solve the evaluation problem. A better-informed labeler still produces fluent labels for incoherent clusters, and without a test nobody can tell. Structure used as input and structure used as a test can be combined. Only the second addresses the gap documented in the Twaddle study.

Why not just feed whole documents to a long-context LLM and let it name themes? That approach yields labels with no corpus-level grouping and the same uncontrolled vocabulary seen with passage-level prompting. It also inherits the position biases reported for flat long-document models in GRETEL, where a graph-topic model beat Longformer-Ext 48.17 to 45.24 ROUGE-1 on arXiv. We include it as a labeler variant scored by $G$ like any other.

Proposed Experiments

The first experiment tests metric validity on Auspol-18. We train LDA and MetaLDA across K values matching the Twaddle setup, plus BERTopic and SenClu. We compute $C_{NPMI}$, $C_V$, LLM rating, and $G$ for every topic and correlate each with expert labeling agreement. If the original annotations cannot be shared, we collect a fresh expert labeling-agreement study on a stratified sample of about 600 topics using the same protocol. We also check whether $G$ picks a K that experts prefer, the question on which $C_{NPMI}$ and $C_V$ disagreed.

The second experiment is a planted-failure test. We build synthetic topics in three ways: mixtures of two real topics' sentences, single-document or single-author dumps, and random sentence sets. All are labeled by the same LLM. We measure detection AUROC for $G$ and for each baseline metric, and check whether the stratum profile correctly separates identity topics (high random, low local) from twaddle (low everywhere).

The third experiment, on the nineteenth-century fiction and Global News corpora from the narrative-labeling study, compares graph-based consolidation with that study's GPT-4o aggregation. Humans judge the merged label sets for redundancy and coverage, and we record the number of LLM calls used.

The fourth experiment, on 20 Newsgroups and on arXiv/PubMed with section trees, tests whether the revision loop improves human preference over FIDELITY's original labels while keeping WE-PD comparable.

The decisive checks are three. $G$ must beat LLM rating in correlation with human agreement. Graph-local negatives must catch planted identity topics that embedding-nearest negatives miss. The revision loop must reduce unsupported content in labels as judged by annotators. If the first check fails, predictive scoring is not worth its cost. If the second fails, the graph contributes nothing and the result should be reported as such.