Large language models can now turn a plain-English description of a scheduling or puzzle problem into a constraint program, meaning code that lists the rules any valid solution must obey and that a solver can execute. The remaining failures are the hard kind. On CP-Bench, about 9% of failures are "logical hallucinations": the model compiles and runs, but it encodes the wrong rules (CP-Bench). Neither execution feedback nor majority voting over samples from one model catches them. This proposal asks several models from different families (Llama, Qwen, Mistral, and so on) to write the constraint program independently. Wherever two of these programs disagree, a constraint solver constructs a concrete assignment that one program accepts and the other rejects. That assignment then becomes a single yes/no question for the user, or for a simulated user: "Is this a valid schedule?"
Constraint programs suit this better than free text. Two text answers can only be compared fuzzily, through embedding similarity. Two constraint programs can be compared exactly: either a distinguishing assignment exists or the programs have the same solutions, and a solver can find that assignment. The proposal therefore joins two literatures that have not met. Heterogeneous ensembling supplies independent hypotheses whose errors are less correlated. Interactive constraint acquisition, the study of learning a constraint model by asking a user membership queries, supplies a sound way to resolve those hypotheses with very few questions. The committee also removes the main bottleneck of interactive acquisition. Normally the space of candidate constraints is enumerated blindly. Here it becomes the small union of rules the models actually proposed.
Core Claim
The claim is that a heterogeneous committee of language models, resolved by solver-generated distinguishing queries at the level of individual constraints, will reach correct constraint models on problems where the best single model with repeated sampling plateaus. It should do so with tens of queries rather than the hundreds or thousands needed by acquisition from an enumerated candidate set. The claim has two separable parts, and the experiments test them separately:
- Distinguishing queries turn model disagreement into correct models cheaply.
- Cross-family diversity is what makes disagreement appear on the confidently wrong cases. A same-model pool would agree unanimously on those cases and so never raise a question.
Relation to Prior Work
Several papers motivate the gap.
- CP-Bench (CP-Bench) benchmarks language-model constraint modeling. It improves reliability with repeated sampling, self-verification and solution majority voting from a single model, reaching 91% with GPT-5.1. It documents both the logical-hallucination residue and a gap between accuracy on a single data instance and accuracy across multiple hidden instances, which arises when models hardcode parameters. We keep its benchmark and harness (kostis-init/CP-Bench). We change two things: the pool is cross-family, and disagreement is resolved by questions rather than votes.
- Cross-model disagreement for uncertainty (Complementing Self-Consistency with Cross-Model Disagreement) shows that disagreement across a jury of 7–9B models from different families flags errors that self-consistency misses. It reports that 2 samples from each of 5 models beat 10 samples from one model. That paper measures disagreement with Sentence-T5 embeddings and concedes that subtle mathematical contradictions can slip past them. Our disagreement measure is exact and comes with a witness assignment, so it can be acted on and not only reported.
- Dependence-aware aggregation (Dependence-Aware Label Aggregation via Ising Models) is a constraint on our design. It shows that language-model judges have correlated errors, so independence-assuming votes become confidently wrong. We treat this as a reason to prefer resolving disagreements with an oracle over aggregating votes. We still run its aggregator as the strongest no-query competitor.
- The diversity ceiling (Ensembling Tabular Foundation Models) is a warning: pools of modern foundation models can be near-redundant. We measure this directly rather than assume diversity.
Other papers supply the method.
- Membership queries. QUACQ-style acquisition and its multi-constraint successor (Multiple Constraint Acquisition) learn constraints from queries. They need 10,110 and 3,821 queries respectively on Sudoku.
- Bottom-up acquisition and learned guidance. GrowAcq (Guided Bottom-Up Interactive Constraint Acquisition) builds the network incrementally. Classifier-guided query generation (Learning to Learn in Interactive Constraint Acquisition) uses a classifier to steer it. We reuse their scope-finding routines in the ActiveConLearn code.
- Language models inside acquisition. LLMACQ (Using LLMs to Improve Query-based Constraint Acquisition) uses one fine-tuned BERT model to parse user explanations, cutting Sudoku to 101.4 queries. HCAR (Overcoming Over-Fitting in Constraint Acquisition) uses an offline XGBoost prior to repair over-fitted global constraints, needing 808 queries on Sudoku.
Both of these use language models or classifiers as priors over an enumerated candidate set. We instead use a committee as the generator of the candidate set itself. That sidesteps the candidate-set explosion that makes QUACQ2 time out on Kakuro.
The agentic routing system AFL (An Agentic Framework with LLMs for Vehicle Routing) uses role-specialized agents from a single model family and outputs heuristics, not verifiable models. It illustrates the same-family blind spot we target, but we do not compete with it.
Research Program
Step 1: generate and decompose. Each of $K$ models from distinct families writes a CPMpy model from the problem text. Each program is broken into its individual constraints. These constraints are lifted to parameterized form where possible, meaning rules written over problem parameters rather than fixed numbers, in the spirit of GENCON (Generalizing Constraint Models). The union of all proposed constraints becomes the candidate set $B$.
Step 2: pick the next question. For each pair of committee models $(M_i, M_j)$, the solver searches for an assignment satisfying $M_i$ but violating at least one constraint of $M_j$. Among the candidate assignments found, we query the one that splits the surviving committee models most evenly. This is query-by-committee carried out with a complete solver rather than by sampling.
Step 3: update from the answer. - A "valid" answer deletes from $B$ every constraint the assignment violates. - An "invalid" answer triggers partial queries on subsets of variables, using the scope-finding routine of the acquisition literature, to localize the missing rule among the candidates it violates.
Step 4: stop and output. The loop stops when no surviving pair of models is distinguishable. The output is the conjunction of surviving constraints, so the final model can mix rules from different committee members.
Exposing hardcoded constants. We generate distinguishing assignments on a second, perturbed data instance. A model that hardcoded a constant then disagrees visibly with one that did not.
When no member has the rule. If the target rule is absent from every member, an invalid answer that no candidate in $B$ explains is fed back to all $K$ models as a counterexample, and they regenerate.
Optimization problems. The objective function is checked separately. Committee models whose optimal values differ produce a pairwise "which of these two solutions is better?" query.
Baselines
Naive reference points. The do-nothing reference is the single best open model in the pool, zero-shot, scored on CP-Bench by its own harness. CP-Bench reports 75% zero-shot for Python frameworks; we will re-measure this for our pool.
Strongest existing method. This is CP-Bench's best configuration: repeated sampling, solution majority voting and self-verification. It reaches 91% accuracy with GPT-5.1 (CP-Bench). We run it with the same total number of model calls as our committee, using cached responses where the harness provides them. Exceeding 91% at matched calls counts as beating it, while also reporting queries used. If closed models are excluded from our pool, we also report the same configuration on our strongest open model. That way the comparison isolates method rather than model strength.
Key ablation: homogeneous committee. This uses identical distinguishing-query machinery on a pool of $K$ samples from one model. It isolates whether heterogeneity matters. The proposal wins this comparison only if the heterogeneous pool reaches higher final accuracy at the same query budget. The direct mechanism to check is that the heterogeneous pool shows disagreement on a larger fraction of the problems that the homogeneous pool gets wrong unanimously.
No-query aggregation. We give the same heterogeneous outputs to two aggregators: independence-assuming weighted majority voting at the solution level, and the class-dependent Ising aggregator (Dependence-Aware Label Aggregation). The Ising couplings are fitted across the 164 problems. On language-model judging tasks the Ising aggregator reaches 0.912 versus 0.820 accuracy on WikiQA relevance. For constraint modeling, no reported result was found on Lacuna. The question this answers is how much of our gain needs an oracle at all.
Classic acquisition benchmarks. On Sudoku, Exam Timetabling and Nurse Rostering, with natural-language descriptions written for each, we compare query counts against three references, all on identical simulated oracles:
- LLMACQ, which reports 101.4 queries on Sudoku.
- HCAR, which reports 808 queries on Sudoku and 100% solution-space recall from 5 examples.
- GrowAcq with classifier guidance, which reports 52% average query reduction.
Why not just…?
Why not just sample many times from the strongest single model? This is the real threat. The query machinery might do all the work, with diversity adding nothing. Hamidieh et al. found that cross-family samples beat same-model samples at equal budget for uncertainty estimation. CP-Bench's 91% plateau suggests that same-model voting leaves confident errors untouched. Neither result is about constraint models resolved by queries, however. The homogeneous-committee ablation settles the question. If it matches the heterogeneous pool, we will report the contribution as the query mechanism alone and drop the diversity claim.
Why not just let a solver or test harness verify the model? A solver checks that a model runs and has solutions, not that it encodes the intended problem. CP-Bench's logical hallucinations pass exactly these checks. Our method does use the solver, but only to build questions whose answers carry information the solver cannot supply.
Why not just run interactive constraint acquisition from scratch? Starting from an enumerated candidate set costs thousands of queries on Sudoku and times out on Kakuro (Using LLMs to Improve Query-based Constraint Acquisition). The committee shrinks that set to tens of rules. The classic-benchmark comparison measures the saving directly.
Why not just measure embedding disagreement and abstain? Embedding-based total uncertainty (Complementing Self-Consistency) flags errors but does not fix them, and it can miss small mathematical differences. We will run it as a detector against solver-witnessed disagreement. Both are evaluated by AUROC for predicting CP-Bench failure, in a no-query mode.
Decisive Checks
The first check is a ceiling analysis. We measure the fraction of CP-Bench problems where some committee member is fully correct, and separately the fraction where the union $B$ contains every target constraint. The union can exceed the first figure because constraint-level repair can combine partly correct models. If either ceiling is barely above the single-model rate, the diversity-ceiling warning applies and the idea fails early, cheaply.
The second check is accuracy across multiple hidden data instances on CP-Bench's 23 multi-instance problems. This tests whether perturbed-instance queries close the hardcoding gap.
The third is a noisy-oracle stress test with 5–10% flipped answers. The omissions work (Omissions in Constraint Acquisition) shows that existing acquisition cannot tolerate wrong answers. We will test whether re-asking only on queries that would delete a constraint proposed by a majority of families restores accuracy.
Success means three results together:
- Accuracy above the matched-budget single-model plateau.
- A clear margin over the homogeneous-committee ablation.
- A median of tens of queries per problem.
References
- CP-Bench: Evaluating Large Language Models for Constraint Modelling
- Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- Multiple Constraint Acquisition
- Guided Bottom-Up Interactive Constraint Acquisition
- Learning to Learn in Interactive Constraint Acquisition
- Using Large Language Models to Improve Query-based Constraint Acquisition
- Overcoming Over-Fitting in Constraint Acquisition via Query-Driven Interactive Refinement
- An Agentic Framework with LLMs for Solving Complex Vehicle Routing Problems
- Generalizing Constraint Models in Constraint Acquisition
- Omissions in Constraint Acquisition