Variational inference turns Bayesian inference into optimization: it searches for a simple distribution that sits close to the true posterior. In collapsed variational inference, the model's global parameters are integrated out analytically before that search begins. In a topic model, these are the per-topic word distributions and per-document topic proportions. What is left is an optimization over per-word topic assignments, and the model sees those assignments only through aggregate counts: how many times each word is assigned to each topic, and how many words in each document go to each topic. This makes the objective smaller and better behaved, but it does not make it convex. The clearest case is the collapsed hierarchical Dirichlet process, a topic model that chooses its own number of topics. It still lands in poor configurations unless it is warm-started from a Gibbs sampler.
This note proposes a split–merge move that works directly on the count matrices of the collapsed model. Collapsing removes the global parameters, and that removal is exactly what makes structural moves cheap to score. Merging two topics amounts to adding two count columns. Splitting a topic amounts to re-dividing only the tokens currently assigned to it. Each move is accepted by comparing an unbiased estimate of a valid collapsed lower bound before and after the move. The CVB0 update itself is not a valid bound, so it cannot serve as the acceptance score. The broader point is that collapsing should be judged as a way to make discrete escape moves cheap and correctly scored, not as a fix for local optima in its own right.
Core Claim
Collapsed Variational Inference for HDP builds the collapsed hierarchical Dirichlet process (CV-HDP), using auxiliary table-count variables and nested truncation so that unused topics are pruned. It also reports that the method is sensitive to initialization, needs a short collapsed Gibbs run to start well, and names collapsed split–merge moves as the open fix. Truly Nonparametric Online Variational Inference for Hierarchical Dirichlet Processes shows that split–merge moves rescue an uncollapsed online HDP (oHDP-SM). Its splits are scored through restricted iterations on explicitly stored global topic parameters, and its merges through topic co-usage covariance. That paper supplies the method template. CV-HDP motivates the gap.
Our claim is that, in the collapsed space, a move's effect on the objective depends only on count statistics. Both proposals and scores therefore need no global parameters and cost time proportional to the tokens touched. A second distinguishing choice is the acceptance score. The widely used CVB0 update discards second-order terms. Rethinking Collapsed Variational Bayes Inference for LDA reinterprets it as a mixed α-divergence projection and shows it beats second-order CVB empirically. Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation notes that CVB0 is not a lower bound on the marginal likelihood. A split–merge scheme that accepted moves by CVB0's surrogate could chase a quantity that does not track model quality.
We avoid that by scoring with the true collapsed evidence lower bound (ELBO) for the factorized $q(z)=\prod_i q(z_i)$:
$$ \mathcal{L}(q) = \mathbb{E}_{z \sim q}\big[\log p(x, z) - \log q(z)\big], $$
where $x$ are the observed words, $z$ the topic assignments, and $\log p(x,z)$ is the closed-form Dirichlet–multinomial marginal computed from the counts of a sampled $z$. Sampling $z$ from $q$ gives an unbiased estimate of $\mathcal{L}$. Using the same random draws for the before-move and after-move states keeps the variance of the difference low.
Research Program
The algorithm alternates ordinary CVB0 sweeps with periodic structural moves.
A merge proposal for topics $k$ and $k'$ is generated from their co-usage across documents, the same signal oHDP-SM uses. It is executed by summing their responsibility columns, so every token's probability mass on $k$ and $k'$ moves to one topic.
A split proposal for topic $k$ takes only the tokens with appreciable mass on $k$. It runs a few restricted CVB0 sweeps that redistribute that mass between two new columns, holding all other topics' counts fixed. The cost scales with the topic's size, not the corpus.
In the HDP case, the top-level topic weights are also collapsed. Here we reuse CV-HDP's auxiliary table-count construction and sample table counts from their exact conditional inside the bound estimate. This keeps $\log p(x, z, t)$ in closed form, where $t$ denotes table counts.
A move is accepted if the paired estimate of $\Delta\mathcal{L}$ is positive with a margin set by its estimated standard error. This guards against accepting noise. We will also derive the deterministic zero-order approximation of the merge score, obtained by plugging expected counts into the log-Gamma terms. The purpose is to measure how often it disagrees with the Monte Carlo score. If the two rarely disagree, the cheaper score suffices. If they often do, that shows CVB0-style surrogates are unsafe for model selection.
The second component is a landscape census that addresses an apparent contradiction in the literature. Fast Variational Inference in the Conjugate Exponential Family shows that the collapsed, KL-corrected bound is less sharply curved than the standard mean-field bound. It reports that Riemannian conjugate gradients on that bound reliably find the global optimum of overlapping Gaussian mixtures, where standard variational Bayes EM (VBEM) fails even after 500 restarts. CV-HDP, by contrast, stays trapped.
Our hypothesis is that collapsing removes continuous traps, the narrow curved valleys in parameter space, but keeps combinatorial traps: two true topics fused into one, or one true topic duplicated. Coordinate ascent cannot leave a combinatorial trap by small steps no matter how flat the bound is. The census runs many random initializations of collapsed and uncollapsed inference and clusters the resulting fixed points by topic matching. It labels each poor fixed point as a fusion, a duplication, or a continuous distortion. The split–merge moves should repair the first two kinds and leave the third untouched.
We will also look at how this varies with document length. The relative-guarantees paper shows the collapsed-vs-uncollapsed bound gap falls below 0.12% on long Yale Law Journal documents but stays up to 2.5% on short ArXiv abstracts. If collapsed inference still reaches better fixed points on long documents, its practical advantage there lies in the optimization landscape, not in bound tightness.
Baselines
The naive reference point is CV-HDP, and CVB0 LDA at fixed $K$, run from random initialization with no warm start. This measures how bad the trapping is when nothing is done about it.
The strongest existing method on this question is oHDP-SM from Truly Nonparametric Online Variational Inference for Hierarchical Dirichlet Processes. It is the only cited method that performs split–merge inside variational inference, and it reportedly recovers the 20 ground-truth bar topics regardless of initialization. That paper gives no scalar perplexity, so on held-out perplexity we have no reported result found on Lacuna for oHDP-SM on NIPS or NYT. It will be run from its published description in both batch and online mode. It gets the same corpora, the same truncation, the same wall-clock budget, and a matched tuning budget over move frequency.
The strongest practical competitor for accuracy is collapsed Gibbs sampling (CGS), together with the hybrid of Hybrid Variational/Gibbs Collapsed Inference in Topic Models. That paper samples tokens with count ≤ 1 by Gibbs and updates the rest with CVB. On KOS it reports test perplexity dropping "from approximately 1800 (CVB) to roughly 1650 (CVB/CGS), matching the gold-standard CGS baseline." Beating CVB means getting below roughly 1800 on KOS. Matching the hybrid and CGS at about 1650 in less time, without any sampling-based warm start, is the bar for success. Exceeding it would be a strong result. On topic count, CV-HDP reportedly finds roughly 113 to 168 topics on KOS. We will report where our method lands and how much that number varies across seeds, since lower variance is itself evidence of escaping initialization dependence.
Two ablations isolate the key ingredients. The first scores moves with the CVB0 surrogate instead of the Monte Carlo collapsed bound, which tests whether a valid score matters. The second runs the identical move schedule on uncollapsed variational inference, re-optimizing global parameters for candidate topics. This tests whether collapsing is what makes the moves cheap and accurate. A third control runs merges only or splits only, to attribute gains to fusion repair versus duplication repair.
Why not just…?
Why not just warm-start from a short collapsed Gibbs run, as CV-HDP already does? This is a real threat, because it is the documented fix. But it is a one-shot repair. Gibbs gives an initialization, after which coordinate ascent can still fuse or duplicate topics as counts evolve, and its cost does not shrink with corpus size the way variational sweeps do. We will compare at equal wall-clock across Gibbs warm-up lengths. If a short warm start matches split–merge in final perplexity and seed-to-seed variance on KOS and NIPS, the proposal reduces to a convenience. We would then report that outcome and drop the efficiency claim.
Why not just run many random restarts and keep the best bound? The Gaussian-mixture result in Fast Variational Inference in the Conjugate Exponential Family, where VBEM failed even after 500 restarts, suggests restarts scale poorly when traps are common. The label-switching benchmark in Globally Convergent Variational Inference, where ELBO methods succeed in only 2–3% of trials, makes the same point. Still, topic models may be easier. We will give restarts exactly our compute budget and choose the winner by the same Monte Carlo collapsed bound.
Why not just anneal the collapsed objective? Deterministic Annealing for Stochastic Variational Inference and Variational Tempering smooth the uncollapsed landscape with a temperature on the entropy term. Tempering CVB0 is easy: one raises responsibilities to a power before normalizing. Our census hypothesis predicts annealing helps with continuous distortions but cannot reliably un-fuse two topics once the temperature is low. We will run tempered CVB0 with a tuned cooling schedule. If it matches split–merge on fusion and duplication counts, the combinatorial-trap hypothesis is wrong.
Why not just use the hybrid Gibbs/variational scheme? The hybrid targets bias from rare tokens, not structural errors in which topics exist. It is complementary: split–merge can run on top of it. We report both alone and combined.
Proposed Experiments
The first experiment is a controlled diagnostic on the synthetic bars dataset, with 2,000 documents and 20 true patterns, as used by oHDP-SM. We run 200 random initializations each of uncollapsed mean-field HDP, CV-HDP, CV-HDP with split–merge, and oHDP-SM. We report the fraction of runs that recover all 20 topics and classify every failure as fusion, duplication, or distortion.
The second experiment is on real text: KOS, NIPS, and 20 Newsgroups with CV-HDP and the HDP variants, and NIPS and Enron for fixed-$K$ CVB0 LDA at $K \in \{40, 80, 120\}$, matching the settings in the Rethinking-CVB paper. We measure held-out perplexity against wall-clock time, final topic count, and seed-to-seed spread. The third experiment is the document-length census on ArXiv abstracts versus Yale Law Journal documents. It measures whether collapsed inference still reaches better fixed points where its bound advantage has vanished.
The decisive checks are three. Split–merge must beat a Gibbs warm start and compute-matched restarts on KOS perplexity and variance. The Monte Carlo collapsed-bound score must accept measurably better moves than the CVB0 surrogate. Most remaining failures under annealing must be combinatorial traps that split–merge repairs. If all three hold, the result is a practical algorithm, plus evidence that collapsing removes continuous traps while discrete moves are needed for combinatorial ones. If the first fails, we learn the trap structure is shallow enough that initialization suffices. That would also be worth reporting.
References
- Collapsed Variational Inference for HDP
- Truly Nonparametric Online Variational Inference for Hierarchical Dirichlet Processes
- Rethinking Collapsed Variational Bayes Inference for LDA
- Relative Performance Guarantees for Approximate Inference in Latent Dirichlet Allocation
- Fast Variational Inference in the Conjugate Exponential Family
- Hybrid Variational/Gibbs Collapsed Inference in Topic Models
- Globally Convergent Variational Inference
- Deterministic Annealing for Stochastic Variational Inference
- Variational Tempering