Relational topic models describe a network of linked documents, such as papers and their citations, with a small set of themes. The model infers each document's mix of topics from its words and from its links, then uses those mixes both to regenerate the words and to predict which documents link to which. Recent neural versions route the citation graph through a graph neural network, a model that updates each document's representation by averaging in its neighbors'. Tuning these models shows a consistent trade-off. Giving the graph more weight improves link prediction and classification, and giving the text more weight makes the topics more coherent. The layer-assisted model LANTM shows this directly with its mixing coefficient $\xi$ (Layer-Assisted Neural Topic Modeling over Document Networks), and GPGBN depends heavily on its balance weight $\beta$ (Deep Relational Topic Modeling via Graph Poisson Gamma Belief Network).
We propose that this trade-off comes largely from the architecture, not from a genuine conflict between text and links. Neighbor averaging acts as a low-pass filter: it keeps what a document shares with its neighborhood and removes what makes it different. That shared part is what predicts links. The removed part is exactly what explains a document's own vocabulary. Over-smoothing, the collapse of node representations toward each other as averaging layers stack, is therefore not only a depth problem here. It pushes the topics toward communities and away from content. Our fix is to split each document's hidden representation at every layer into a neighborhood-shared band and a document-specific band. The link decoder reads only the shared band. The word decoder reads both. If the hypothesis holds, one model should match the best text-weighted setting on topic coherence and the best graph-weighted setting on link prediction simultaneously, without a global weight to tune.
Core Claim
The existing work treats text and graph as two modalities to be blended, by a scalar in LANTM and by a likelihood weight in GPGBN. Our claim is that blending should happen by graph frequency and not by modality. Text features enter at every layer, as in LANTM's per-layer fusion. The graph then decomposes those features into a smooth part and a residual part, and the two decoders are wired to different parts.
Let $\hat A$ be the symmetrically normalized adjacency matrix with self-loops, and let $H^{(l)}$ be the document representations after layer $l$ (text MLP output fused with the previous layer). Each layer computes the two bands and the topic mixes:
$$ H_S^{(l)} = \hat A H^{(l)},\qquad H_D^{(l)} = (I-\hat A)H^{(l)},\qquad \theta^{\text{link}}_d=\mathrm{softmax}(W_S h^{(L)}_{S,d}),\quad \theta^{\text{word}}_d=\mathrm{softmax}(W_S h^{(L)}_{S,d}+W_D h^{(L)}_{D,d}). $$
Here $h_{S,d}$ and $h_{D,d}$ are document $d$'s rows of the shared band $H_S$ and the difference band $H_D$, and $W_S$, $W_D$ project each band into topic space. Because $W_S$ is shared across both decoders, the two topic mixes live in the same topic space, and the link topics stay a coarsening of the word topics. Links use the relational-topic-model link function $p(y_{ij}=1)=\sigma(\eta^\top(\theta^{\text{link}}_i\circ\theta^{\text{link}}_j)+\nu)$ from Relational Topic Models for Document Networks, where $\circ$ is the elementwise product, $\eta$ is a learned per-topic weight vector, and $\nu$ is a learned bias. Words come from a standard neural-topic-model decoder applied to $\theta^{\text{word}}$.
The split $I=\hat A+(I-\hat A)$ is borrowed from the filterbank GNN of Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks. That paper adds a high-pass "diversification" channel and mixes the two channels with learned scalars for node classification. The new ingredient is the routing: which decoder is allowed to see which band. The filterbank is only the tool that makes routing possible. A document with no links has $\hat A h = h$ up to normalization, so the difference band vanishes and the model reduces to a text-only encoder. This preserves the cold-start behavior that Relational Topic Models for Document Networks emphasized.
Relation to Prior Work
Three neural relational topic models motivate the proposal, and each leaves the smoothing problem in a different place.
LANTM fuses an MLP text channel with a plain GCN channel at every layer through $\xi$. Its GCN channel is purely low-pass, and its reported TC-versus-F1 trade-off across $\xi$ is the phenomenon we aim to explain.
GPGBN pairs a deep Poisson Gamma belief network with a GCN-based Weibull encoder. Its standard convolutions are equally low-pass.
GRTM (Graph Relational Topic Model with Higher-order Graph Attention Auto-encoders) sidesteps over-smoothing differently. It precomputes shortest-path proximities, attends over them with a log-normal decay prior, and finds that a single attention layer works best. That result constrains our design. Adding depth to a low-pass encoder is known to hurt in this setting, so any multi-layer claim must show that depth helps once the difference band is preserved. Otherwise we should report that GRTM's shallow answer is sufficient.
Two other over-smoothing remedies fix the problem inside the encoder, for classification. The adaptive gate in When Do GNNs Work: Understanding and Improving Neighborhood Aggregation switches aggregation off per node using neighborhood entropy and center-neighbor similarity. The band-pass channel attention in Geometric Scattering Attention Networks selects frequency responses per node. Neither considers a model with two decoders that want different frequencies, which is our case.
Work on late fusion supplies the reason not to fuse only at the end. Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models documents modality laziness, where a jointly trained model leans on the dominant input and neglects the other. ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning analyzes its coupled message passing spectrally, showing that the learned coupling amplifies cross-modal agreement modes and attenuates disagreement modes. ReCoG has no interpretable topics and no difference channel, and it discards the "disagreement" signal that we argue carries document-specific vocabulary.
A measurement caution comes from Are we measuring oversmoothing in graph neural networks correctly?, which shows that similarity- and energy-based metrics conflate useful smoothing with collapse. We therefore measure the harm of smoothing through its effect on held-out word likelihood, not through representation distance alone.
Research Program
We will build the model as a variational autoencoder, an encoder-decoder trained to reconstruct its inputs through a probabilistic latent space. It uses a logistic-normal topic prior, the per-layer text injection of LANTM, the band split above, and the routed decoders. Depth varies from one to eight layers. Held-out test edges are removed from $\hat A$ before encoding, so link scores cannot leak through the smoothing.
The analysis has three parts.
First, we test whether the trade-off is caused by smoothing. For LANTM, a plain GCN relational topic model, and our model, we trace topic coherence and held-out word perplexity against link-prediction AUC. LANTM's frontier is traced over $\xi$. Our model's frontier is traced over a single optional reweighting of the link loss, used only for this diagnostic.
Second, we test where the gains come from. We stratify test edges by center-neighbor text similarity and by whether they cross label communities. If routing works as claimed, link gains should concentrate on boundary edges and gains in word likelihood on boundary documents.
Third, we test whether depth becomes useful. We plot word perplexity and AUC against depth for the plain GCN and routed variants, alongside the per-layer Dirichlet energy of $\theta^{\text{word}}$ reported with the caveat above.
Baselines
The naive reference point is a text-only neural topic model with no graph, together with the classic two-stage pipeline of fitting topics and then running a logistic regression on topic products. Relational Topic Models for Document Networks reports that joint RTM improves citation-suggestion precision by 80% over this pipeline on Cora. We rerun both so that every method is measured against the same floor.
The original RTM, with a text-only encoder and links entering only through the likelihood, is the no-smoothing baseline. It is the most important sanity check, because it never averages over neighbors at all. We implement it as a neural variational model with the same decoders, so that only the encoder differs.
LANTM is the closest existing method, and its $\xi$ trade-off is our target. We tune $\xi$ over the grid its authors used, plus a finer grid, under the same budget of trials we give ourselves. The file reports its absolute numbers only for classification (Macro-F1 of 0.688 on the Cora PL subset versus 0.554 for AdjEnc-X) and a relative link result (AUC improved by about 0.160 over text-only models on the ML subset). We will reproduce both before comparing.
GRTM is the strongest over-smoothing-aware relational topic model. No reported absolute AUC or PMI value for GRTM was found on Lacuna; its results appear only in figures. We run it from the authors' description, tuning the log-normal variance $\sigma$ with the same budget.
GPGBN is the strongest reported link predictor. Deep Relational Topic Modeling via Graph Poisson Gamma Belief Network reports 96.5% AUC on Citeseer, which sets the bar there.
All methods share data splits, vocabulary preprocessing, topic count, encoder width, and ten seeds.
Two ablations isolate the key ingredient. The first, "filterbank without routing," replaces LANTM's GCN with the two-channel filterbank but lets both decoders read the learned mix, as in FB-GNN. The second, "routing without per-layer injection," keeps the band routing but feeds text only at the input. Beating the baselines means that, at the topic coherence of LANTM's best-coherence setting, our link AUC matches or exceeds LANTM's best-AUC setting across seeds, and that we are at least on par with GPGBN's Citeseer AUC and with GRTM on the Cora subsets.
Why not just…?
Why not just swap LANTM's GCN for a filterbank GNN? This is the obvious combination and the most serious threat, which is why it is our first ablation. The filterbank paper's learned channel mix optimizes one objective. With two decoders pulling in opposite directions, we expect the mix to settle at a compromise, reproducing the $\xi$ trade-off in learned form. If the unrouted filterbank reaches the same frontier as the routed model, the routing claim fails, and we will say so.
Why not just keep the encoder shallow, as GRTM does? GRTM's finding that one layer is best is strong evidence that low-pass depth hurts. It does not show that depth is useless once the difference band is kept. Our depth sweep settles this. If the routed model also peaks at one or two layers and does not beat GRTM, then shallow encoding is the answer, and the contribution shrinks to a better explanation of the trade-off.
Why not just gate aggregation per node, as ALaGCN does? A per-node gate in the style of When Do GNNs Work, which achieved 73.3% versus 59.2% for GCN on Cora with 1% labels, still produces a single representation shared by both decoders. Closing the gate on a boundary document helps its words but removes the neighborhood signal its links need. We include an ALa-gated LANTM as an additional baseline to test this directly.
Why not just avoid smoothing entirely, as the original RTM does? RTM's text-only encoder is a real competitor. Layer-Assisted Neural Topic Modeling over Document Networks reports roughly 0.160 AUC gain over text-only models, which suggests that graph encoding matters for links. Its authors did not separate out whether the gain comes at a cost in coherence, and our frontier plot does.
Proposed Experiments
The main experiments use the four Cora subsets (DS, HA, ML, PL), full Cora, Citeseer, and PubMed for link AUC, topic coherence measured by PMI, held-out perplexity, and document classification Macro-F1. We add Reuters R8 with its text-similarity k-nearest-neighbor graph, where smoothing should be nearly harmless because edges are built from text.
The stress test is WebKB, which gives both regimes. RTM used it for hyperlink prediction, and its Texas and Cornell pages are the heterophilic graphs, where linked pages often belong to different classes, on which the filterbank paper reports roughly 10% classification gains. There, our model should beat LANTM and GRTM by the widest margin on cross-class links and on word perplexity.
Three decisive checks follow. The central result would be a frontier of coherence against AUC that dominates LANTM's across seeds; a frontier that merely matches it would refute the central claim. The routing ablation must separate from the unrouted filterbank. Gains must concentrate on low-similarity edges. If they are spread uniformly, the improvement is ordinary capacity and not frequency routing.
References
- Layer-Assisted Neural Topic Modeling over Document Networks
- Deep Relational Topic Modeling via Graph Poisson Gamma Belief Network
- Relational Topic Models for Document Networks
- Complete the Missing Half: Augmenting Aggregation Filtering with Diversification for Graph Convolutional Networks
- Graph Relational Topic Model with Higher-order Graph Attention Auto-encoders
- When Do GNNs Work: Understanding and Improving Neighborhood Aggregation
- Geometric Scattering Attention Networks
- Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models
- ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning
- Are we measuring oversmoothing in graph neural networks correctly?