Topic models summarize a document collection as a handful of themes, each shown to people as a list of its most probable words. Those lists often fail at the level of single words. "Kernel," "attention," or "transformer" mean different things in different subfields, and a topic model that treats each spelling as one symbol tends to merge unrelated themes or split one theme into odd fragments. Many real collections, such as papers, patents, and product pages, come with links between documents: citations, hyperlinks, co-purchases. A paper's cited neighbors say which "kernel" it means far more reliably than its short title does. This proposal uses those links to split ambiguous words into senses before the topic model runs. The topic model then works over "kernel (operating systems)" and "kernel (methods)" as distinct tokens, and its top-word lists should become more interpretable.
Two sentences capture the idea. We encode each occurrence of a word with a language model that sees both the document's own text and short text from its graph neighbors, cluster those occurrences into senses, and hand any existing topic model a corpus rewritten in sense tokens. The bet is that document links resolve word ambiguity that a document's own text cannot, especially when that text is only a title, and that this shows up as topics that humans and model judges find more coherent. The bet is not simply that the tokens differ.
Core Claim
On collections of documents connected by links, which the graph literature calls text-attributed graphs, graph-conditioned word-sense splitting should raise human-judged and model-judged topic coherence over the same topic model run on raw words. It should also beat sense splitting from the document's own text alone, and the gain should be largest on short texts and on words that are ambiguous across subfields. The contribution is a representation layer that sits in front of a topic model. It is not a new topic model, so it can be tested inside LDA, a corpus-statistics prior, and a contrastive neural topic model, with the model held fixed.
The motivating evidence comes from both directions. On the topic side, word-sense ambiguity is a known cause of poor topic labels; Labeling Results of Topic Models: Word Sense Disambiguation as Key Method for Automatic Topic Labeling with GermaNet maps topic words to entries in a hand-built lexicon, and static embeddings are known to collapse domain senses into one vector (Measuring Topic Coherence through Optimal Word Buckets, which scores coherence by grouping topic words from their embedding similarities). On the graph side, GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph nests neighbor aggregation inside each Transformer layer. It argues, and shows on link prediction, that neighbor text disambiguates a node's own text. No one has pointed that disambiguation at the vocabulary of a topic model. We also replace the curated lexicon with the corpus's own link structure.
Relation to Existing Bridges
Several papers already join graphs and topic models, and the proposal is defined by what each leaves out. Graph Contrastive Topic Model runs graph convolutions over word co-occurrence and document-word graphs to mine better contrastive samples. GINopic: Topic Modeling with Graph Isomorphism Network encodes a word-similarity graph inside each document. Both use only word-level graphs and treat documents as unlinked, and both keep one vocabulary entry per spelling. Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks shows that adding knowledge-graph structure to language-model embeddings improves coherence; its purely language-model variant is clearly worse. That result motivates our claim that structure adds something text embeddings lack. TEC, however, only works when documents contain entities listed in Wikidata, while we take structure from the corpus's own links and keep ordinary vocabulary. Graph-Structured Topic Modeling for Documents with Spatial or Covariate Dependencies (GpLSI) uses a document graph to smooth each document's topic mixture toward its neighbors. It changes which topics a document gets but never changes what a word means, which makes it the most important alternative to test against. Hierarchical Graph Topic Modeling with Topic Tree-based Transformer (GTFormer) nests citation-graph and hyperbolic topic-tree embeddings in every Transformer layer. It is the closest joint model of text, links, and topics, and it is expensive; the file reports classification and link-prediction results for it, not coherence. Our approach is deliberately lighter and orthogonal: a preprocessing step that any of these models could consume.
Research Program
The pipeline has three stages. First, for each document we build a short neighborhood context from the titles of a small sample of its graph neighbors, with five neighbors following GraphFormers' sampling budget. We encode every occurrence of each candidate word with a frozen sentence encoder applied to the document text together with that context. The primary variant simply appends neighbor titles to the input. A second variant uses GraphFormers-style fusion of cached neighbor embeddings. If the simple variant matches the fused one, we will report that and recommend it.
Second, for each word type above a frequency floor, we cluster its occurrence embeddings with a Gaussian mixture, which PRISM also uses to cluster word embeddings for its Dirichlet prior. The number of components is chosen by a held-out likelihood criterion, and we accept a split only when the resulting clusters are better separated than those from the text-only encoding of the same occurrences. This rule means a word is split only when the graph adds disambiguating signal, so unambiguous words stay intact and the vocabulary does not grow without cause.
Third, we rewrite the corpus in sense tokens and run unchanged topic models on it: MALLET-style LDA, PRISM, and GCTM. For display, each sense token receives a two-word gloss drawn from its most distinctive co-occurring words.
The analysis examines where gains come from. We bin topic words by a cross-community ambiguity score, defined as the entropy of the graph communities their occurrences fall in. We also compare title-only against full-abstract inputs, since the prediction is that graph conditioning matters most when a document's own text is short, the setting where co-occurrence statistics are known to collapse.
Baselines
The naive reference point is each topic model on the raw word-type corpus, with the same preprocessing, number of topics, and iteration budget. Every other comparison is reported as a difference from this.
The strongest existing coherence method in the file is GCTM, which reports NPMI 0.379 on 20 Newsgroups with 50 topics, against 0.353 for SCHOLAR+BAT, both in Graph Contrastive Topic Model. We rerun GCTM with its published configuration on each corpus, raw and sense-tokenized, at matched topic counts and five seeds. Beating it means a statistically reliable gain in model-judged intrusion accuracy and CTC rating on the linked corpora, with NPMI no worse than that of GCTM on raw words. There is no reported result found on Lacuna for topic coherence on ogbn-arxiv, Cora-ML, or DBLP for any model, so those numbers must come from our own reruns.
GpLSI is the strongest method that uses the document graph without splitting words. We run it on the same citation graph with its graph cross-validation for the smoothing parameter. Beating it means higher coherence at comparable held-out likelihood. We also run a combination of GpLSI and sense tokens to test whether the two effects add up.
GTFormer is the strongest joint text-graph-topic model. We run it where compute allows (Cora-ML, DBLP) with the same topic count, and report its wall-clock cost alongside ours. The claim here is comparable coherence at much lower cost, not necessarily higher coherence.
Three ablations isolate the key ingredient. Text-only sense induction clusters the same occurrences without neighbor context. Shuffled-graph sense induction uses neighbors from a degree-preserving random rewiring. Random pseudo-senses split each word into the same number of tokens by random assignment, which tests whether vocabulary inflation alone moves the metrics. The method has to beat all three for the claim to stand.
Why not just…?
Why not just use a contextual language model on each document's own text to induce senses? This is the most direct competitor and is exactly our text-only ablation. On full abstracts it may well suffice, because an abstract usually disambiguates itself. Our prediction is narrower: on titles and other short texts, own-text context is too thin, and neighbors carry the missing signal. If text-only induction matches graph-conditioned induction on title-only ogbn-arxiv, the core claim is false, and we will say so.
Why not just smooth document topic mixtures over the graph, as GpLSI does? GpLSI borrows strength from neighbors for short documents, which is part of what we want. It still assigns one topic-word distribution per spelling, so an ambiguous word pulls two themes together within a topic's word list regardless of how well the mixtures are smoothed. This is a real threat on corpora where ambiguity is rare. The ambiguity-binned analysis decides it: if our gains over GpLSI do not concentrate on high-entropy words, the mechanism is not what we claim.
Why not just feed frozen language-model document embeddings through a graph network and cluster them? A cascaded pipeline yields good document clusters but no word-level topic descriptions. GraphFormers reports that cascaded language-model-plus-graph pipelines trail nested fusion by 2.9% to 6.5% on DBLP link-prediction metrics, which is evidence that neighbor-blind encoding loses information. We nonetheless include this pipeline, with topic words extracted by class-based term weighting over its clusters, because clustering embeddings is a strong and common alternative.
Why not just trust NPMI? Splitting words changes co-occurrence counts, so NPMI computed on the rewritten corpus is partly circular. Contextualized Topic Coherence Metrics documents co-occurrence metrics being fooled by frequent but meaningless "trash topics." Our primary metrics are therefore its language-model intrusion and rating scores plus a human word-intrusion study. NPMI is computed only after collapsing sense tokens back to surface words against an external reference corpus, and glosses are removed before any judge sees the lists, so the glosses themselves cannot inflate the scores.
Proposed Experiments
The linked corpora are ogbn-arxiv (about 170k computer-science papers with citations), run in both title-only and title-plus-abstract form, together with Cora-ML and DBLP. 20 Newsgroups, which has no graph, is a negative control. There our method reduces to text-only induction and should give no graph-attributable gain, and GCTM's reported 0.379 NPMI anchors our reproduction.
The decisive check is a comparison on title-only ogbn-arxiv at 50 and 100 topics. Graph-conditioned sense tokens must beat text-only sense tokens, shuffled-graph sense tokens, and random pseudo-senses on intrusion accuracy, and the margin must grow with the ambiguity score of the topic words. A second check hand-annotates occurrences of about fifty ambiguous words sampled from ogbn-arxiv and measures sense-clustering agreement for graph-conditioned against text-only encodings. This verifies that splitting follows actual meaning rather than citation cliques. A third check tests whether the benefit survives inside all three topic models, which would make it a property of the representation and not an artifact of one model. The idea fails if the shuffled graph performs as well as the real one, or if random pseudo-senses capture most of the gain.
References
- Labeling Results of Topic Models: Word Sense Disambiguation as Key Method for Automatic Topic Labeling with GermaNet
- Measuring Topic Coherence through Optimal Word Buckets
- GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph
- Graph Contrastive Topic Model
- GINopic: Topic Modeling with Graph Isomorphism Network
- Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks
- Graph-Structured Topic Modeling for Documents with Spatial or Covariate Dependencies
- Hierarchical Graph Topic Modeling with Topic Tree-based Transformer
- PRISM: PRIor from corpus Statistics for topic Modeling
- Contextualized Topic Coherence Metrics