Delphi on Disputed Cases: Inducing Verifiable Rules When Examples Underdetermine Them

Language models increasingly write their own rules, either as small programs that explain a handful of input–output examples or as symbolic checks that filter their own answers. Both uses break in the same place. A few examples almost never pin down one rule, so many candidate rules fit the evidence and then diverge on inputs nobody has checked. On list-function tasks, Analysis of Error Sources in LLM-based Hypothesis Search for Few-Shot Rule Induction found that nearly half of the programs passing held-out tests were still wrong against the true rule. It also found that its summarization step, where a language model condenses 64 candidate explanations into 8, throws away the correct rule about 26% of the time. In domains where the missing rule is background knowledge that experts never write down, such as "the reaction slot of a commonsense relation must be a mental state, not an event," the examples say even less.

We propose to borrow the Delphi method, a survey protocol in which experts rate items anonymously, see the group's aggregate ratings and the dissenters' written reasons, and re-rate until the ratings stabilize. We apply it not to the rules themselves but to the specific inputs where candidate rules disagree. First, candidate rules are executed to find these "disputed cases." Then an anonymous panel of language-model raters judges the correct behavior on each case over several rounds, with minority reasons carried forward instead of summarized away. Finally, the rules are scored by how well they match the settled cases. The output is an executable rule set: programs for few-shot induction, and relational constraints that replace the hand-written ones in a verification plugin.

Core Claim

When examples underdetermine a rule, the missing information has to come from a prior. In professional and commonsense domains that prior is domain knowledge. Our claim is that structured, multi-round deliberation over disputed cases extracts this prior more reliably than the two things current systems do.

The first is lossy summarization of hypotheses, as in the Parab et al. pipeline. The second is picking whatever behavior most candidate programs agree on. That second option is a real competitor, and the proposal stands or falls on beating it.

The specific new ingredient is the Delphi retention rule. A disputed case is settled only when the panel's agreement is stable across rounds. A dissenting behavior survives to the next round if its rationale is not refuted, even when it starts in the minority. We predict two things. Rules chosen this way will have a lower false-pass rate (passing tests but wrong against the ground-truth rule) than rules chosen by majority execution behavior. Induced constraints will recover most or all of the gain that hand-written constraints give in the ConstraintChecker setting.

Research Program

The system has three stages.

Generation. We keep the generation stage of Parab et al.: 64 natural-language hypotheses, each implemented as a program or, for relational tasks, as a typed constraint over a relation's head and tail. There is no summarizer. Every hypothesis is implemented, which directly removes the 26% discard channel.

Finding disputed cases. We execute all candidates on automatically generated probe inputs. For list functions these are random lists. For commonsense relations they are unlabeled triples from the knowledge base. We keep the probes where candidates split into two or more behavioral clusters. Each such probe becomes a Delphi item whose options are the competing behaviors.

Deliberation. A panel of rater agents, drawn from different model families and prompts, judges each item. In round one each rater independently picks a behavior, with a short rationale and without seeing other raters. In later rounds raters see the distribution of votes and the anonymized rationales, with minority rationales always shown. They may revise their vote or rebut. Consensus uses the percentage bands from the ankle-cartilage consensus of Dombrowski et al.: strong consensus at 75–99% agreement. Stopping is defined by vote stability between rounds rather than a fixed round count. Items that never stabilize are marked unresolved and carry no weight.

Scoring. Each candidate rule gets a Delphi score: the fraction of resolved disputed items on which it produces the consensus behavior, plus its accuracy on the given examples. The top-scoring rule is returned.

For the relational setting, the accepted constraints are compiled into the plug-in verifier of ConstraintChecker. That system checks a base model's plausibility judgment against per-relation typing and temporal constraints and combines the two with a logical AND. Its constraints were written by hand, and its authors name automatic constraint induction as open work. We keep their architecture and replace only the constraint source. This makes the comparison clean.

Two prior results set limits on the design.

Rules must be enforced, not just shown. TheoremQA found that pasting the relevant theorem into the prompt gave no significant gain. ConstraintChecker similarly reports that rules placed inside the main prompt interfere with it. So we never hand induced rules back to the model as text. They are always executed as external filters.

Consensus is not truth. Dombrowski et al. stress that Delphi consensus stands in for evidence rather than replacing it. We therefore evaluate every Delphi output against ground truth, never against the panel's own agreement.

We deliberately do not use differentiable rule learners such as Neuro-Symbolic Hierarchical Rule Induction as the core. They need clean, pre-parsed relational facts, and our inputs are free text.

Baselines

Do nothing. The naive reference point is direct generation with no search.

  • On list functions, Parab et al. report 0.359 mean test accuracy for direct GPT-4o generation.
  • On CKBPv2 and SD-ATOMIC20, it is the base model's zero-shot plausibility judgment with no verifier. The research file gives no absolute F1 for this, so we will measure it ourselves.

Strongest search method. On list functions the strongest method is the Parab et al. hypothesis search, with 64 hypotheses, 8 summarized rules, and up to 3 debugging passes. It reports 0.487 mean test accuracy against a human average of 0.521, using up to 257 model calls per task. We rerun it on the same 100 tasks with the same base model. Our system is capped at the same call budget, with deliberation calls counted in that budget. We also report the ground-truth correctness of every returned program, not only test accuracy.

Beating it requires two things. Test accuracy must rise above the rerun number. The false-pass rate must drop. The file reports only "nearly half" for that rate, so we compare against our own rerun measurement.

Strongest verifier. On the commonsense benchmarks the strongest method is ConstraintChecker with its hand-written constraints. It reports about 1% average F1 gain on CKBPv2 and up to 2.32% F1 gain for ChatGPT zero-shot on SD-ATOMIC20. No absolute F1 values are given in the file. We use the same base models, prompts, and aggregation rule. Matching the hand-written gain with fully induced constraints would count as success. Exceeding it would mean the panel found constraints the authors missed.

Ablations isolating deliberation. Three ablations separate the Delphi ingredient from its surroundings:

  • No summarizer. Implement all 64 hypotheses and select by example accuracy alone. This isolates how much comes simply from removing the lossy summarization step.
  • Execution majority. Settle each disputed probe by the behavior most candidates produce, with no raters.
  • One-round panel. Take a single blind vote and never show rationales.

A fourth ablation drops anonymity. Rationales are labeled with the model that wrote them, so we can test whether raters defer to a nominally stronger model. This is the machine analogue of the status bias that Delphi was designed to remove.

Why not just…?

Why not just skip the summarizer and implement all 64 hypotheses? This removes the 26% discard problem at no deliberation cost, and it may account for much of any gain. That is why it is an explicit ablation. It does nothing about the other half of the problem, though: programs that fit every given example yet generalize wrongly. The prediction is that it improves recall of the correct rule but leaves the false-pass rate roughly unchanged. If it also fixes the false-pass rate, deliberation is unnecessary on list functions, and we will say so.

Why not just take the majority behavior on disputed inputs, with no deliberation? This is the most serious threat. It is cheap, it uses execution, and it needs no raters. Its weakness is that candidates generated by the same model share its biases, so the majority behavior tends to be the most common misreading rather than the correct rule. The execution-majority ablation tests this directly. The decisive quantity is the set of disputed items where the round-one minority becomes the final consensus. We will measure how often those reversals match ground truth. If reversals are no more accurate than chance, the Delphi machinery adds nothing, and the proposal reduces to execution voting.

Why not just ask one strong model to state the constraints, or retrieve them from a handbook? TheoremQA's null result for theorem augmentation shows that having a rule in text does not mean it gets applied. Error analysis in the medical reading-comprehension study attributes 44% of failures to knowledge missing from both text and knowledge graph, so static sources also miss coverage. A single-model constraint writer is still a fair baseline. We will run it: one call per relation, with its output compiled into the same verifier. We report its F1 gain next to both the panel's and the hand-written constraints'.

Proposed Experiments

Mechanism test on list functions. The first experiment uses the 100 list-function tasks, where ground-truth rules are known. For every method we report:

  • mean test accuracy;
  • ground-truth correctness rate among test-passing programs;
  • how often the correct rule survives to final selection;
  • model calls per task.

The decisive check has two parts. At matched call budget, the full system must lower the false-pass rate relative to both the no-summarizer and the execution-majority ablations. Minority-to-consensus reversals must be right more often than they are wrong.

Application to commonsense verification. The second experiment induces per-relation constraints for the ATOMIC-style relations, using training triples as examples and unlabeled triples as probes. We compile them into ConstraintChecker and evaluate F1 on CKBPv2 and SD-ATOMIC20. We compare against hand-written constraints, single-model constraints, and no verifier. We also inspect the induced constraints against the authors' hand-written taxonomy of typing, temporal, and ambiguity constraints. ConstraintChecker's ablation found typing constraints mattered most, so we check first whether those are recovered.

Calibration and stability. Across both settings we plot, for each resolved disputed item, the final agreement level against ground-truth correctness. This tests whether stable high agreement predicts truth or merely shared error, the gap between consensus and evidence that Dombrowski et al. warn about.

A natural extension, left for after these checks, is the appellate setting of AppealCase. There, all 20 evaluated models stay below 50% F1 on reversal prediction, and disputed cases would be first-instance records on which induced reversal rules disagree.