Teaching Time-Varying TF-IDF to Agree with What the Encoder Actually Uses

Embedding-based topic pipelines first group documents by meaning, then name each group with keywords. The naming step uses class-based TF-IDF, a word score that rewards words frequent inside a cluster and rare in the rest of the collection. When the collection spans decades, the "rare in the rest" half becomes ambiguous. Rare compared with which period: this year, the last decade, or all of history? A word like network was diagnostic in 1990s machine-learning papers and is background vocabulary today. A word like speech has stayed stable for thirty years. If the reference window is fixed, one of these two kinds of word gets mislabeled. That error appears downstream as keyword lists that jump around between years, or as lists that keep showing stale words long after they stopped carrying meaning.

We propose to let each word choose its own reference window for the inverse-document-frequency term, mixing frequencies measured over horizons of 1, 2, 4, 8… time slices. We fit that choice against an external signal: gradient-based saliency, meaning the derivative of a document's topic-assignment score with respect to its input tokens, computed with a method that stays on real word embeddings. Saliency is too expensive to run over a whole stream, so we compute it on a small sample. A lightweight gate over each word's recent frequency history then learns to reproduce it, and that gate runs cheaply over the full corpus. Put briefly, the expensive attribution teaches the cheap lexical statistic which timescale to use, word by word.

Core Claim

A per-word, data-dependent mixture of temporal horizons in the IDF term will produce dynamic topic keywords that are smoother across time and more faithful to the clustering model than two alternatives: per-slice c-TF-IDF, and the single best-tuned exponential-decay IDF. The gain should be concentrated on corpora that mix long-lived and bursty vocabulary. The claim is falsifiable. If a tuned single half-life matches the adaptive mixture, the adaptive part has no value, and we will report that result.

The motivating evidence comes from three places. Online multiscale dynamic topic models (MDTM) showed that topic word distributions built from weighted mixtures of dyadic historical horizons beat single-scale dynamic models on perplexity, and that they separate evergreen terms (speech, recognition) from transient ones (HMM). MDTM does this inside a Gibbs-sampled bag-of-words generative model and never touches term discriminativeness or neural encoders. We move the same multiscale intuition into the IDF step of an embedding pipeline. Evaluating Dynamic Topic Models supplies both the problem and the yardstick. It shows that snapshot coherence metrics reward erratic transitions, and it introduces Temporal Topic Coherence (TTC), Temporal Topic Smoothness (TTS), their product Temporal Topic Quality (TTQ), and Dynamic Topic Quality (DTQ). Of these, TTS correlates with human judgments at Spearman's ρ up to 0.83 on NeurIPS. Unified Neural Topic Model via Contrastive Learning and Term Weighting (UTopic) supplies the closest use of cluster-based TF-IDF inside a neural topic model. There, the weights scale an autoencoder's reconstruction loss. UTopic's weights are static and are never checked against what the encoder relies on. Our proposal changes both properties.

Research Program

The base pipeline is a standard embedding topic model. Sentence embeddings are reduced with UMAP and clustered with HDBSCAN, and each cluster's documents are bucketed by time slice $t$. For word $w$ and dyadic scale $s \in \{0,\dots,S-1\}$, let $p_s(w,t)$ be the fraction of all documents in the window of $2^s$ slices ending at $t$ that contain $w$. The temporal IDF is

$$\text{IDF}_t(w) = -\log \sum_{s=0}^{S-1} \lambda_s(w,t)\, p_s(w,t),$$

where $\lambda_s(w,t) \ge 0$ are mixture weights summing to one. A small gate produces these weights from features of $w$'s own frequency history: the recent values of $p_s(w,t)$, their ratios across scales, and a burstiness statistic. This design follows the context-dependent scale gating of Time-Aware Multi-Scale RNNs for Time Series Modeling and the Gumbel-Softmax scale routing of Learning to Adaptively Scale Recurrent Neural Networks. Neither of those methods has been applied to term statistics. The topic keyword score is the usual within-cluster term frequency multiplied by $\text{IDF}_t(w)$.

The teaching signal works as follows. For a sampled document $d$ at time $t$ assigned to topic $z$, we define its assignment margin. This is the cosine similarity of its embedding to $z$'s centroid at $t$, minus the log-sum-exp of its similarity to all other centroids at $t$. Because the score depends on contrast, a word only receives credit if it separates this topic from the others in the current period. We attribute the margin to input tokens with Discretized Integrated Gradients (DIG). DIG integrates gradients along a path that hops between real vocabulary embeddings rather than through meaningless interpolated points. It reported a better deletion log-odds than plain Integrated Gradients on SST2 with DistilBERT (−1.259 vs −0.950). Summing token attributions per word across a topic's sampled documents gives a saliency profile $A_{z,t}(w)$. The gate parameters $\phi$ are then fit with a listwise objective that makes the keyword score distribution match the saliency distribution:

$$\min_\phi \sum_{z,t} \mathrm{KL}\!\left(\mathrm{softmax}(A_{z,t}/\tau)\;\|\;\mathrm{softmax}(\text{TF}_{z,t}\cdot \text{IDF}^{\phi}_t/\tau)\right),$$

where $\tau$ is a temperature and $\text{TF}_{z,t}$ the within-cluster term frequency.

The program has three parts. First, we build the gate and teacher, and we fix the sampling budget, for example 2% of documents per slice. Second, we test whether a gate fit on early epochs or on one corpus transfers, without new saliency, to later epochs and to other corpora. This transfer property is what makes the method cheap enough for streams. Third, we analyze the learned $\lambda$. If the method works as hypothesized, words that MDTM-style analyses call evergreen should load on long horizons, and bursty terms should load on short ones.

Baselines

The do-nothing reference is per-slice c-TF-IDF, with IDF computed only within slice $t$, alongside its opposite, global c-TF-IDF over the whole corpus. Both run on the identical embeddings and clusters, so only the keyword step differs. Beating them means higher TTQ and DTQ on held-out slices together with a higher deletion-faithfulness score (defined below).

The strongest simple competitor is exponential-decay IDF with a single half-life. Its half-life is tuned on the same validation slices, with the same number of configurations we spend tuning the gate. This baseline directly tests whether one well-chosen timescale suffices. To beat it, the adaptive mixture must exceed it on TTQ and faithfulness on at least the mixed-dynamics corpora, with the difference larger than seed-to-seed variation over five clustering seeds.

The strongest existing dynamic topic models are MDTM, D-ETM (The Dynamic Embedded Topic Model, whose topics follow a single-step Gaussian random walk in embedding space), and the LLM-based D-LLM evaluated in Evaluating Dynamic Topic Models. We run each with its authors' recommended settings on the same slices and vocabulary. D-ETM reported training in under six hours on UN debates versus nearly two days for D-LDA, which bounds the compute we allot. No reported result was found on Lacuna for TTQ or DTQ of D-ETM, D-LDA, or D-LLM on NeurIPS, UN debates, or NYT; that file reports only the qualitative finding that D-LLM trails the specialized models. We therefore rerun all of them ourselves.

Three ablations isolate the key ingredients. In the first, the mixture weights are fixed and uniform across scales, which tests whether adaptivity matters or just averaging. In the second, the gate is fit by MDTM-style next-slice predictive likelihood of word frequencies instead of by saliency, which tests whether the saliency teacher matters. In the third, the teacher is attention-pooled token weights from the encoder instead of DIG, which tests whether principled attribution matters.

Why not just…?

Why not just tune one decay half-life? This is the real threat. If corpora are dominated by one dynamic regime, a single half-life will be close to optimal. MDTM's qualitative split between evergreen and transient terms suggests otherwise for long, heterogeneous corpora like NeurIPS and State of the Union, but that is suggestive, not proof. The tuned-half-life baseline above settles it. We expect the mixture to win on corpora that span decades with mixed dynamics, and possibly to tie on the 23-day Digg stream, and we will report either outcome.

Why not just fit the scales by likelihood, as MDTM does? Likelihood rewards predicting which words appear. It does not reward predicting which words distinguish a topic from its neighbors, and the latter is what a keyword list needs. The likelihood-fit ablation tests this directly. If it matches the saliency-fit gate on deletion faithfulness and TTQ, the saliency component is unnecessary, and the paper becomes a simpler multiscale-IDF result.

Why not just use attention weights, which come free from the forward pass? Attention is not Explanation found attention correlates weakly with gradient importance (mean Kendall's τ ≈ 0.34 on SST with BiLSTMs). It also found that very different attention distributions can leave predictions essentially unchanged. The elephant in the interpretability room argues for saliency over attention on these grounds. The attention-teacher ablation checks whether this unreliability carries over to our setting.

Why not just run saliency on every document and skip TF-IDF? Path-based attributions need tens to hundreds of backward passes per document, which is prohibitive for streams of the size of NYT. The transfer experiment shows whether a cheap gate preserves most of the saliency-ranked quality. We also report full-saliency keywords on the smallest corpus as an upper bound.

Proposed Experiments

We use the longitudinal corpora from the file. These are NeurIPS 1987–2019, UN General Debates 1970–2020, and NYT 1987–2007 from Evaluating Dynamic Topic Models, plus State of the Union (205 epochs), PNAS titles (91 epochs), and Digg (23 daily epochs) from MDTM. Together they cover century-scale, decade-scale, and day-scale dynamics.

Topic keyword quality is measured with NPMI, topic diversity, TTC, TTS, TTQ, and DTQ. Faithfulness is measured with an attribution-free deletion test, which keeps the evaluation separate from the training signal. For each topic and slice, we delete its top-$k$ keywords from held-out documents and measure the drop in assignment margin, comparing against deleting $k$ frequency-matched random words. Larger drops mean the keywords really drive the clustering. We also run the deletion test on DIG's own rankings to verify that the teacher deserves trust.

Three checks are decisive. The first asks whether adaptive mixture IDF beats the tuned single half-life on held-out-slice TTQ and deletion faithfulness on NeurIPS and State of the Union. The second asks whether the saliency-fit gate beats the likelihood-fit gate on faithfulness. The third asks whether a gate trained on NeurIPS transfers to UN debates while keeping most of its in-domain gain. As a secondary qualitative check, we inspect whether the learned scale weights place known evergreen and bursty terms on long and short horizons respectively.