Probabilistic circuits are generative models built as graphs of sums and products. They can answer exact probability questions, such as the likelihood of a partially observed image or the probability of one region given another, in time linear in the graph's size. Their weakness has been scale. The approach that made them competitive on natural images, latent variable distillation, takes features from a large pretrained network, clusters them with K-means, and uses the cluster labels as hidden variables for the circuit. Those teacher features are high-dimensional and highly redundant, and in pretrained vision features the directions of largest variance tend to encode ordinary background variation rather than the structure that matters. K-means on raw distances follows variance. So the circuit's hidden states may be spent separating backgrounds from each other.
We propose to select which teacher feature directions the clustering is allowed to see. The selection signal is the student circuit's own exact held-out likelihood, which needs no labels. This turns a supervised trick from anomaly detection into an unsupervised one. That trick is greedily picking scattered, low-variance eigencomponents because they perform better. The swap is possible only because the student is tractable: its likelihood is an exact, cheap score for any candidate subspace. The point of the proposal is that a circuit's exactness can be used to remove redundancy from the representation it is distilled from, not only to answer queries at the end.
Core Claim
The latent variable distillation pipeline of Scaling Up Probabilistic Circuits by Latent Variable Distillation clusters embeddings from BERT or a Masked Autoencoder (a vision transformer pretrained by reconstructing hidden image patches). The cluster labels become discrete hidden variables, which splits training of a large circuit into independent sub-circuits and breaks the plateau where big circuits stop improving. The authors themselves call K-means on embeddings a heuristic. They note that splitting data across clusters makes each sub-circuit data-hungry, so clusters that duplicate each other are directly costly. Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits analyses the student–teacher gap as a distillation error plus the teacher's variational gap. Its progressive-growing variant (LVD-PG) reassigns clusters and grows structure during training, reaching 4.06 bits per dimension on ImageNet32. Neither paper asks which part of the teacher's representation the clusters should be built from.
Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection shows that for pretrained CNN features, the top principal components carry normal background variation. Their detector reaches 100% AUROC with fewer than 10 eigencomponents on MVTec-AD categories such as bottle and wood, and those components are scattered non-consecutively across the spectrum. Adding high-variance components makes it worse. That method needs labeled defects to score each candidate subset. This paper motivates our hypothesis that teacher variance is a poor guide. It also supplies the search procedure we reuse, greedy bottom-up and top-down selection over eigencomponents. Our change is the scoring function.
Concretely, let $U$ be the eigenvectors of the teacher-embedding covariance, $S$ a subset of their indices, and $z_S(x)$ the K-means assignment of example $x$ computed on the projection of its embedding onto $U_S$. Let $p_{\theta_S}$ be a student circuit trained with those assignments as observed hidden variables. We choose
$$S^\star = \arg\max_{S} \; \frac{1}{|D_{\text{val}}|}\sum_{x \in D_{\text{val}}} \log p_{\theta_S}(x),$$
where $D_{\text{val}}$ is held-out data and the log-likelihood marginalizes the hidden variables exactly, as the circuit permits. Because cluster assignments are fixed, each sub-circuit trains independently. That keeps a cheap low-capacity proxy student fast enough to sit inside a greedy loop. We then train the full-size student once on $S^\star$.
Research Program
The work has three parts.
The first is diagnostic. On ImageNet32 and CIFAR-10 with Masked Autoencoder teachers, we measure how redundant standard LVD clusters are. We count pairs of clusters whose sub-circuits assign near-identical likelihoods to each other's data, which the circuit computes exactly. We also check how much of the between-cluster separation lies in the top principal directions. If clusters are not redundant and do not track background variance, the hypothesis fails early and cheaply.
The second part builds the selection procedure. The proxy student is the same circuit family at roughly one-tenth width. We search over eigencomponents in blocks of adjacent indices, then refine single components, which keeps the number of proxy fits in the low hundreds. We report the rank correlation between proxy and full-size likelihood across about twenty subsets, because the method is only as good as that correlation. The same procedure applies to text, using BERT embeddings on WikiText-2.
The third part tests what the selection buys beyond bits per dimension, using queries only a tractable model can answer exactly. We measure conditional log-likelihood of masked image regions given the visible rest, under center, half-image and random-patch masks. We also run unsupervised anomaly detection on MVTec-AD: train the circuit on normal images only and score test images by exact likelihood. The MVTec-AD test closes the loop with Gula and Bertoldo. If label-free, likelihood-chosen components overlap with the components their labeled search finds, that is direct evidence the circuit's likelihood recovers task-relevant structure from a redundant representation.
Two neighbouring lines constrain the design. HyperSPNs generates circuit weights from a small network and beats weight decay on 18 of 20 discrete benchmarks. Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits shares one neural network across hundreds of circuit units, cutting parameters from 262M to 387k. Both remove redundancy among the circuit's parameters. Neither touches redundancy in what the hidden variables encode, so they are compatible and could be layered on top. Priors for Diversity in Generative Latent Variable Models replaces independent priors with a determinantal point process, which penalizes similar components. In their K-means variant this spreads CIFAR-10 centroids about 14% further apart and improves features. It is the closest alternative fix for redundant clusters, and we treat it as a baseline rather than a component. On the Relationship Between Probabilistic Circuits and Determinantal Point Processes shows that encoding such repulsion inside a deterministic circuit generally needs exponential size. This is a reason to apply anti-redundancy at the distillation stage, outside the circuit, rather than as a circuit-level prior.
Baselines
The naive reference point is a circuit trained without distillation. Here that is the HCLT circuit, a hierarchical Chow-Liu tree, which Scaling Up Probabilistic Circuits by Latent Variable Distillation reports at 4.82 bpd on ImageNet32. Any distillation variant should clear this easily, so it mainly calibrates our reimplementation.
The direct competitor is standard LVD: the same teacher, K-means on the full embedding, the same number of clusters K and the same circuit architecture. That paper reports 4.39 bpd on ImageNet32. We rerun it under our code with identical teacher checkpoint, K, circuit size, optimizer and epochs. Beating it means a lower bpd than our rerun by more than the spread across three seeds, with the search's proxy compute reported separately and also matched by giving the baseline an equal extra training budget.
The strongest existing method is LVD-PG from Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits, at 4.06 bpd on ImageNet32. Because it reassigns clusters during training, it could undo whatever a better initial clustering provides. We therefore run LVD-PG twice, starting from full-embedding clusters and from $S^\star$ clusters, with identical growing schedules. The claim survives only if the $S^\star$ start ends below the standard start, not merely below 4.39.
Two ablations isolate the key ingredient, which is the likelihood-based score rather than dimensionality reduction as such. The first keeps the same number of components $|S^\star|$ but takes the top-variance ones. The second keeps the same number but takes a random subset. For CIFAR-10 and WikiText-2 no reported result was found on Lacuna for LVD or LVD-PG bpd or perplexity, so all comparisons there are against our own reruns. For MVTec-AD, the reference is the 100% AUROC with fewer than 10 components on bottle and wood reported by Gula and Bertoldo. That number uses labels, so it is an upper reference for our method, not a bar to beat.
Why not just…?
Why not just whiten the teacher embedding before K-means? Whitening puts every direction on equal footing, which removes the variance dominance we blame. It is the most serious threat to this proposal. It differs from our method in one respect: it keeps noise directions at full weight instead of discarding them. Gula and Bertoldo's degradation curve, where adding components hurts, suggests that keeping all directions is harmful even once their scale is equalized. That evidence is from a Gaussian detector, not from circuits. We run whitened K-means as a full baseline at matched K. If it matches $S^\star$, the honest conclusion is that the problem is scale rather than selection, and we will report it that way.
Why not just project onto the top principal components? This is the standard redundancy fix. The anomaly-detection evidence argues it keeps exactly the wrong directions. Our top-variance ablation at equal dimension tests this directly in the distillation setting.
Why not just use more clusters, or the determinantal-point-process K-means of Zou and Adams? More clusters fragment the data further, and LVD already identifies fragmentation as its sample-efficiency problem. Repulsion pushes centroids apart, but in a background-dominated space it may push them apart along background directions. We run DPP-K-means with its repulsion weight tuned on validation likelihood under the same budget as our search. Our prediction is that it reduces duplicate clusters as measured by the diagnostic, yet improves likelihood less than subspace selection. Should it win, the distillation stage needs repulsion rather than selection, which would still validate redundancy as the bottleneck.
Why not just rely on LVD-PG's dynamic reassignment? Reassignment may repair a poor start. The paired LVD-PG runs described under Baselines settle this directly.
Decisive Checks
Four results decide the proposal.
The first is whether $S^\star$ excludes most top-variance components while beating both the equal-size top-variance subset and the random subset in held-out bpd on ImageNet32. If selected components are just the top of the spectrum, the hypothesis is wrong.
The second is whether the gain over standard LVD survives progressive growing. Gains that vanish under LVD-PG would mean the method only fixes what growing already fixes.
The third is whether conditional likelihood of masked regions improves at least as much as joint bpd. This would show that the hidden variables now carry structure that exact conditional queries use.
The fourth is whether, on MVTec-AD, the unsupervised selection overlaps the components that label-guided greedy selection picks, and whether likelihood-based AUROC rises relative to a full-embedding LVD circuit.
A positive result on the first and fourth would show that exact likelihood can stand in for labels when pruning redundant teacher features. A negative result on the first, especially if whitening matches us, would retire the selection story cleanly.
References
- Scaling Up Probabilistic Circuits by Latent Variable Distillation
- Understanding the Distillation Process from Deep Generative Models to Tractable Probabilistic Circuits
- Gaussian Image Anomaly Detection with Greedy Eigencomponent Selection
- HyperSPNs: Compact and Expressive Probabilistic Circuits
- Scaling Continuous Latent Variable Models as Probabilistic Integral Circuits
- Priors for Diversity in Generative Latent Variable Models
- On the Relationship Between Probabilistic Circuits and Determinantal Point Processes