Live Topic Sets: Unbounded On/Off Topic Chains with Generalized Dirichlet Weights

Topic models summarize a document collection as a mixture of themes, but most of them fix the number of themes in advance. Fields change: themes are born, merge, and die, and inside any one document a handful of themes co-occur in ways that are not independent. Current models handle at most two of these three facts. Some track how topics correlate over time but keep the number of topics fixed. Some let the number of topics grow but have no notion of time. Some track an unbounded set of on/off processes over time but cannot represent proportions that sum to one.

We propose a model in which each year of a corpus is described by a binary vector saying which topics are alive. Each topic's on/off status follows its own small Markov chain, and the number of chains is unbounded. Within a document, the weights over the live topics are drawn from a generalized Dirichlet distribution, a relative of the Dirichlet that is built from a chain of independent Beta "stick-breaking" fractions and has more freedom in how components co-vary. The key design choice is the split of labor. The unbounded binary chains decide how many topics exist and when. The generalized Dirichlet acts only on the finite set of currently live topics. This removes the variance-shrinkage machinery that infinite correlated topic models need, and that machinery is what suppresses niche topics. The weights also stay conjugate to word counts, so inference can integrate them out in closed form.

Core Claim

A sequential topic model can have three properties at once without the costs each one carries on its own. The first is a distributed state: the "state" of year $t$ is a binary vector over topics, so $M$ chains give $2^M$ configurations, rather than a single categorical variable. The second is unbounded cardinality: the number of chains is inferred. The third is non-Dirichlet weights: correlations among topic proportions are not forced negative. This works if cardinality is handled by a temporal Indian Buffet Process over topic existence, and the generalized Dirichlet is applied only to the live subset. Because that subset is finite in every year, the generalized Dirichlet needs no $1/k$ shrinkage. Because the generalized Dirichlet is conjugate to multinomial counts, the cost of adding or removing a topic in a year reduces to a ratio of Beta functions. The model makes three testable predictions. It should match or beat a fixed-$K$ dynamic correlated topic model on held-out perplexity. It should detect topic births and deaths that fixed-$K$ models smear across existing topics. It should keep rare-but-locally-dominant topics that shrinkage-based nonparametric models suppress.

Relation to Prior Work

Three prior works each cover two of our three ingredients.

The Infinite Factorial Hidden Markov Model introduced the Markov Indian Buffet Process: an unbounded set of binary Markov chains with slice-sampling inference. It applied this to source separation, and its chains gate continuous emissions rather than proportions. Infinite Factorial Dynamical Model extended it to continuous states with particle MCMC, again for physical signals. We reuse their prior over chains (they supply the method), but apply it at the corpus-year level. There each chain has only $T\approx 29$ steps, so forward-filtering backward-sampling is cheap and the scaling concerns those papers raise largely disappear.

Hidden Markov Models Based on Generalized Dirichlet Mixtures for Proportional Data Modeling showed that generalized Dirichlet emissions beat Dirichlet emissions on correlated proportional sequences. It decomposes the distribution into independent Betas ("W-space"). Its hidden state is a single categorical variable, and both $K$ and the number of mixture components are fixed by hand. We borrow the Beta decomposition and place it inside a distributed, unbounded state.

The Doubly Correlated Nonparametric Topic Model achieves unbounded, correlated topics through a logistic stick-breaking process. It needs a $1/k$ variance shrinkage prior to keep total variance finite, it is static, and it uses non-conjugate Metropolis-within-Gibbs. The IBP Compound Dirichlet Process and its Application to Focused Topic Modeling motivates our decoupling. It shows that tying a topic's global prevalence to its local weight dilutes niche topics, and it fixes this with a binary topic selector. Its authors note that the weights of selected topics are independent Gammas, so topics cannot co-vary. Our model can be read as a dynamic focused topic model whose selected-topic weights are generalized Dirichlet rather than independent. That is the closest prior design, and our ablations isolate exactly that difference.

Online Learning of a Dirichlet Process Mixture of Generalized Dirichlet Distributions for Simultaneous Clustering and Localized Feature Selection pairs generalized Dirichlet with nonparametrics, but as a single-assignment static clustering model. Stochastic Variational Inference for Dynamic Correlated Topic Models tracks evolving topic covariance with Gaussian and Wishart processes at scale, but with $K$ fixed. It is our strongest competitor.

Model

Topics $k = 1, 2, \dots$ each have a word distribution and a binary chain $z_{tk}\in\{0,1\}$ over years $t$. The chains follow the Markov Indian Buffet Process: topic $k$ switches on with probability $a_k$ and stays on with probability $b_k$, with stick-breaking priors that order topics by popularity. Let the live set in year $t$ be $A_t = \{k : z_{tk}=1\}$, ordered $k_1 < \dots < k_m$. A document $d$ in year $t$ draws its weights by generalized Dirichlet stick-breaking over the live topics only:

$$ \theta_{d,k_i} = V_{d,k_i}\prod_{j<i}\left(1 - V_{d,k_j}\right),\qquad V_{d,k}\sim \mathrm{Beta}(\alpha_k,\beta_k), $$

Here $\theta_{d,k_i}$ is the weight of the $i$-th live topic in document $d$, and $V_{d,k}$ is the fraction of the remaining stick that topic $k$ takes. The last live topic takes the remainder. The per-topic pair $(\alpha_k,\beta_k)$ is global, so a topic keeps its characteristic weight behavior whenever it is alive.

Given token counts $n_{dk}$, the posterior of each $V_{d,k}$ is $\mathrm{Beta}(\alpha_k + n_{dk},\ \beta_k + \sum_{j>k,\ j\in A_t} n_{dj})$. As a result, the document weights integrate out in closed form. The marginal likelihood of all year-$t$ documents under a proposed live set is then a product of Beta-function ratios, and this quantity is exactly what is needed to resample $z_{tk}$.

One limitation must be stated up front. The covariance a generalized Dirichlet can express depends on the ordering of components, and it is more restricted than a full logistic-normal covariance. We treat the ordering as a modeling choice to study. We compare popularity order, birth-time order, and an order learned by Metropolis swaps of adjacent topics. We do not assume the distribution captures every correlation that a correlated topic model finds.

Research Program

We will build a collapsed Gibbs sampler with four moves. The first resamples token-topic assignments using the closed-form generalized Dirichlet predictive. The second resamples each topic's year chain with forward-filtering backward-sampling, using the Beta-ratio marginals as emission terms. The third uses slice sampling to bound the number of instantiated topics, following the iFHMM. The fourth updates $(\alpha_k,\beta_k)$ with Newton steps in the Beta decomposition, following HMMGD.

We will analyze two things about the generalized Dirichlet itself. The first is what correlation patterns it can and cannot represent once restricted to a moving live set. The second is whether removing shrinkage causes runaway topic creation. The Markov Indian Buffet Process prior should already control this, since its expected number of live topics is finite.

A secondary instantiation tests generality beyond text. It treats each video frame's optical-flow histogram as a generalized Dirichlet mixture of live "motion primitive" chains.

Baselines

The naive reference point is LDA fit independently per year with a single $K$ chosen by held-out perplexity over a grid. It has no dynamics, no correlation, and no cardinality inference, and it is what most practitioners would do.

The strongest existing method is the dynamic correlated topic model of Stochastic Variational Inference for Dynamic Correlated Topic Models. On the NeurIPS 1987–2015 corpus it reports per-word perplexity of approximately 505, against roughly 874 for the next best dynamic model. Splits and vocabularies are not specified in the file, so we will rerun it with its released configuration on our own splits and preprocessing. We will sweep $K$ over the same grid used for LDA, with the same wall-clock budget. Beating it means lower held-out per-word perplexity than its best $K$ on our splits. Matching it while also recovering topic births it misses would count as a partial win.

Two static baselines isolate the role of dynamics. The Doubly Correlated Nonparametric Topic Model and The IBP Compound Dirichlet Process and its Application to Focused Topic Modeling will be run on the pooled corpus and per year. Both report beating HDP only through curves. No reported scalar result was found on Lacuna for DCNT or FTM perplexity, so we will produce these numbers ourselves. HDP is included as the standard nonparametric reference.

Three ablations carry the main argument, and each swaps exactly one ingredient while keeping the sampler and budget fixed. The first replaces the generalized Dirichlet over live topics with a symmetric Dirichlet over live topics. This is effectively a dynamic focused topic model, and it isolates the value of the generalized Dirichlet. The second replaces it with a logistic normal over live topics, which isolates generalized Dirichlet versus Gaussian correlation. The third replaces the Markov chains with an independent per-year Indian Buffet Process, which isolates temporal persistence.

On the proportional-video side, the reference is HMMGD on Weizmann. The file reports about 55% accuracy for baseline HMMGD, about 73% with an optical-flow prior, and 91.9% rank-2 accuracy, with Dirichlet HMMs lower.

Why not just…?

Why not just run the dynamic correlated topic model with a large fixed $K$ and let unused topics fade? This is the most serious threat. A large-$K$ Gaussian-process model can drive a topic's mean weight near zero in early years, which imitates a birth. Our experiments settle it in two ways. We plot its perplexity against $K$ and check whether it degrades past the optimum. We also plant synthetic topic births into real years and measure the year at which each model first assigns the planted topic distinct words. If large-$K$ DCTM detects births as sharply and matches our perplexity, the cardinality component adds little for corpus modeling, and we will report that.

Why not just put a logistic normal over the live set instead of a generalized Dirichlet? A logistic normal expresses arbitrary covariance, while the generalized Dirichlet does not. The file's evidence for the generalized Dirichlet concerns conjugacy and parameter count ($2K-2$ versus $O(K^2)$) and comes from HMMGD on proportional features, not from text. The live-set logistic-normal ablation is the direct test. If it wins on perplexity at comparable cost, the generalized Dirichlet's contribution reduces to speed. We will measure that speed honestly as time per sweep and time to a fixed perplexity.

Why not just use a plain Dirichlet over the live set? The binary selector already decouples prevalence from local weight, so the extra flexibility might be unnecessary. HMMGD's synthetic result, 89.7% versus 80.1% when features are positively correlated, suggests the Dirichlet's forced negative covariance does hurt. That result, however, comes from emissions rather than topic weights. The Dirichlet-over-live-set ablation answers this. We will also report the fitted $(\alpha_k,\beta_k)$, to show whether the learned weights actually depart from the Dirichlet special case.

Why not just use one HMM whose states are topic configurations? With $M$ live-able topics this needs up to $2^M$ states and a squared number of transition parameters, as Factorial Hidden Markov Models shows. In the iFHMM source-separation experiments, the non-factorial alternative also found 9 speakers instead of 5. We include it only at small $M$ as a sanity check.

Proposed Experiments

The primary benchmark is the NeurIPS 1987–2015 corpus with yearly slices and held-out documents within each year. The headline metric is per-word perplexity against the baselines above. We add the NIPS 1987–1999 corpus used by DCNT for the static comparison.

The first decisive check is the planted-birth test described above, scored by detection delay and false-birth rate. The second is a niche-topic test. We inject a small set of documents dominated by a rare synthetic topic and measure the weight each model assigns it within those documents. Our model and FTM should keep that weight high, while DCNT and HDP should shrink it.

The third check is a correlation-recovery test. We generate data from a correlated topic model with known positive correlations and measure how much of that structure each weight model recovers under each ordering scheme. We expect the generalized Dirichlet to fall short of the logistic normal here, and we want to measure by how much.

The fourth check is the Weizmann action-recognition replication against HMMGD's roughly 73%. It tests whether a distributed state of motion primitives beats HMMGD's single categorical state on the same optical-flow histograms.

The proposal fails cleanly in two cases: if the Dirichlet-over-live-set ablation matches the full model on every test, or if large-$K$ DCTM matches it on both perplexity and birth detection.