Pooling Proofs Across Different Models to Keep Top-k Neuro-Symbolic Training from Locking In Early Mistakes

Neuro-symbolic systems train a neural network to recognize low-level facts, such as "this image is the digit 7" or "this sentence says A is B's mother," using only a label for a downstream logical conclusion. The network gets no fact-level labels at all. Computing how likely a conclusion is requires summing over every chain of facts (every "proof") that could produce it, and that sum is exponential in size. Practical systems such as Scallop therefore keep only the $k$ most probable proofs and backpropagate through those. The shortcut has a failure mode that has not been studied directly. The $k$ proofs are chosen by the network's current beliefs. If the network starts out wrong, the correct proof never enters the retained set, receives zero gradient, and the network is trained to prefer the wrong facts that did make the cut. The retained proofs behave like pseudo-labels the model assigns to itself, so top-$k$ training inherits the confirmation bias familiar from self-training.

We propose to let several perception networks with different architectures share a single proof pool. At each step, every network contributes its own top-$k$ proofs. The union of those proofs is the set over which all networks compute the query probability and receive gradient. A proof that a CNN ranks 40th but a vision transformer ranks 1st stays in play for the CNN, which gets a direct corrective signal it could never generate alone. The claim is specific. When the models' errors are uncorrelated, a pooled set of $M \cdot k$ proofs contains the correct proof far more often than one model's top-$(M \cdot k)$ proofs. This higher "proof recall" is what lets training recover from a bad start.

Core Claim and Mechanism

Consider an example with query $q$ and $M$ perception models. Let $T_j$ be the $k$ highest-probability proofs of $q$ under model $j$, and let $U = \bigcup_j T_j$ be the pooled set. Model $m$ is trained on

$$\hat P_m(q) = \sum_{\pi \in U} \prod_{f \in \pi} p_m(f),$$

where $\pi$ ranges over proofs in the pool, $f$ over the base facts a proof uses, and $p_m(f)$ is model $m$'s probability for fact $f$. The only change from Scallop's differentiable top-$k$ proof mode is that the summation set comes from the whole committee rather than from model $m$ alone. Each model still uses its own probabilities, so no one is forced to copy another model's hard answer.

The diagnostic quantity is proof recall: the fraction of training examples whose true proof lies in the retained set. It is measurable on benchmarks where the hidden fact labels are known, such as the digit identities in handwritten-formula tasks. The prediction follows from the gradient argument above. Final accuracy under a poor initialization should track proof recall during the first epochs. Pooling should raise that early recall by more than matched-size single-model truncation does, and the gain should shrink as the members' errors become correlated.

Relation to Prior Work

The method builds on Scallop, which attaches differentiable provenance tags to Datalog derivations and bounds them to the top-$k$ proofs to make otherwise intractable (#P-hard) inference practical (Scallop: From Probabilistic Deductive Databases to Scalable Differentiable Reasoning; Scallop: A Language for Neurosymbolic Programming). Marra et al. push the same idea to $k=1$ and report up to 1,000x speedups when probability mass is concentrated on one explanation (Approximate Inference for Neural Probabilistic Logic Programming). Both treat the perception front end as a single model trained from whatever starting point it has. Neither asks what happens when mass is concentrated on the wrong explanation early in training, which is the case we target.

The motivation comes from two literatures. Work on iterative refinement repeatedly finds that a bad initial seed places the true solution outside the refinement's search window. SharpContour's misplaced contours and IEBins' collapsed depth intervals are examples (SharpContour; IEBins), and the top-$k$ pool is exactly such a window. Work on self-training shows that a model trained on its own confident mistakes enters a "death spiral." For example, GPT-4o self-refinement on Green Patent falls from about 68% to about 61% across iterations (Self Iterative Label Refinement via Robust Unlabeled Learning).

The heterogeneity ingredient is supported by evidence that architecturally different models fail differently. Pairing a CNN with a vision transformer lets each fix the other's pseudo-labels in semi-supervised segmentation (Transformer-CNN Cohort), and debates among different LLM families avoid the echo chamber of same-model debate (ReConcile). Two results constrain the design. Foundation-model ensembles can hit a "diversity ceiling" where different architectures still make redundant predictions (Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap). A much weaker member can drag a committee down (MAMM-Refine). We therefore measure error correlation among members directly and test for member-parity effects rather than assuming them away.

The closest existing work is Ambiguity-aware Ensemble Training for Semi-supervised Dependency Parsing. It merges the 1-best trees of three structurally different parsers into a forest and trains a CRF parser on that forest with exact dynamic programming, reaching 93.19% versus 92.34% unlabeled attachment score on the Penn Treebank. That is a one-shot annotation pipeline whose structures are directly observed parse trees. Our setting differs in three ways. The shared objects are latent proofs of a logical query, the pool is rebuilt at every step from models that are themselves being trained, and the question is whether pooling changes the long-run dynamics of truncated training, including whether the members converge toward one another over time. That convergence question is left open by both that paper and Transformer-CNN Cohort.

Research Program

We will implement proof pooling inside Scallop's top-$k$ provenance, as a union of per-model proof sets followed by per-model re-scoring. We will then study it along three axes.

The first axis is controlled initialization. We pretrain each perception model on a small labeled subset with a controlled fraction of permuted labels. This sets initial fact accuracy on a grid from near-chance to good, which lets us map where single-model top-$k$ training fails to recover and whether pooling moves that boundary.

The second axis is committee composition. We compare heterogeneous committees against same-architecture committees differing only in seed and augmentation. The heterogeneous committees combine a CNN, a vision transformer, and a prototype classifier on frozen self-supervised features. For each committee we log pairwise fact-error correlation over training. This tests whether diversity persists or the models homogenize as they train on one another's proofs.

The third axis is parity. We deliberately include one much weaker member and compare unweighted pooling against pooling where each model's contribution to the pool is capped by its current agreement with the query labels.

Baselines

The naive reference point is Scallop's standard differentiable top-$k$ proof mode with a single CNN perception model at its default small $k$, trained end to end from the same corrupted initializations. The strongest reported number for this family on HWF is Scallop maintaining "over 92% accuracy even when training data was reduced to 25%" (Scallop: A Language for Neurosymbolic Programming). We treat this as the reference to reproduce under clean initialization before any comparison. For MNIST-R accuracy, Pathfinder, and CLEVR there is no reported result found on Lacuna for Scallop. CLUTRR has only a relative figure: Scallop is about 25% better than fine-tuned GPT-3.

The strongest competitor is a single model with a proportionally larger truncation, $k' = M \cdot k$. Its retained set is the same size as the pooled set, so it costs comparable reasoning work. This baseline directly tests whether the benefit comes from pool size or from pool diversity. The second strong competitor is a single model trained for $M$ times as many steps, or with a parameter count matched to the whole committee, at the original $k$. We also run the $k=1$ variant of Marra et al. as the extreme-truncation reference.

Two ablations isolate the key ingredient. The first is a same-architecture committee using the identical pooling rule, which isolates heterogeneity from ensembling. The second is a heterogeneous committee that exchanges only each member's single best proof as a hard target, which is ordinary co-training with pseudo-labels. It isolates the value of pooling proofs while letting each model score them with its own probabilities.

All methods get the same data, the same labeled-subset initializations, and an identical tuning budget over learning rate and $k$ (or $k'$) on a held-out split. They are matched on total training FLOPs and on the maximum retained-set size per example. Pooling counts as beating the baselines only under three conditions. It must reach higher final accuracy than matched-size $k'$ and matched-compute single models on the low-initialization half of the grid, averaged over at least five seeds with non-overlapping intervals. Its early-epoch proof recall must be higher. And it must not lose accuracy in the good-initialization regime.

Why not just…?

Why not just increase $k$? This is the most serious threat, and the design is built to settle it. If the true proof sits at rank 20 under the CNN, a single model with $k'=M \cdot k$ may recover it without any committee. Our bet rests on correlated misranking. Under a systematically wrong initialization, the true proof is often at a rank far beyond any affordable $k'$, while a model that fails differently ranks it near the top. The proof-recall curves at matched pool size will show which case holds. If larger $k$ matches pooling on both recall and final accuracy across the initialization grid, we will conclude that truncation diversity is not the bottleneck and report that as the finding.

Why not just ensemble copies of one architecture with different seeds? This is cheaper and simpler. It is weakened by evidence that same-architecture members fail together: CNN-CNN pairs share local blind spots in Transformer-CNN Cohort, same-model debate degenerates in ReConcile, and an 18-member same-architecture ensemble in Auto-Annotation Quality Prediction still needed a separate learned filter to catch pixels the members got wrong together. The diversity ceiling result cuts the other way, since architectural difference does not guarantee uncorrelated errors. The homogeneous-committee ablation, together with the logged error correlation, decides whether heterogeneity or mere multiplicity carries the effect.

Why not just warm up the perception model or filter by confidence? Better supervised pretraining would move the initialization out of the failure regime. Our controlled-corruption grid treats initialization quality as the independent variable, so it shows how much pretraining quality pooling is worth. Confidence filtering fails exactly where we operate, because an overconfident model's wrong proofs are its most confident ones (Approximate Bayes Optimal Pseudo-Label Selection). Its Bayesian alternative has so far been shown only on small tabular problems. We include a confidence-gated single-model variant, which weights examples by the top proof's share of retained mass, as an additional baseline.

Proposed Experiments

The primary benchmark is HWF, with MNIST-R as a second perception-heavy suite. On both, digit and operator labels allow exact computation of proof recall. On each, we sweep initial fact accuracy and report final query accuracy, fact accuracy, proof recall over the first epochs, and member error correlation over training for every baseline and ablation above.

The first decisive check is a correlation test. Across runs, early proof recall should predict final accuracy better than initial fact accuracy does. If it does not, our account of the failure is wrong.

The second decisive check is the matched-size comparison against $k' = M \cdot k$. If pooling does not help here, the idea does not work.

The third check tracks error correlation over training to see whether heterogeneous members converge. If they do, pooling should help early and fade later. That outcome would call for periodically re-diversifying the members, for example by resetting part of one model.

As a transfer test, we run CLUTRR with text relation extractors from different model families feeding Scallop's kinship rules. We train on short chains and evaluate on longer ones, where the number of possible proofs grows quickly and truncation bites hardest. We add the weak-member parity experiment on HWF to test whether capped pool contributions prevent the degradation reported by MAMM-Refine.