Joint Soft-Logic Selection of Self-Generated Knowledge

When a language model is asked to write down relevant facts before answering a question, it usually answers better. It also inherits every false "fact" it writes. In Generated Knowledge Prompting for Commonsense Reasoning, a frozen model generates several background statements per question and then answers conditioned on them. This raised zero-shot accuracy by 7 to 10 points. Yet 86% of the remaining wrong answers were traced directly to hallucinated background statements. The statements are produced and scored one at a time, so nothing checks them against each other.

We propose to treat the model's own background statements the way classical NLP treated noisy local classifier outputs: as variables in a small joint optimization with soft constraints. The model is prompted, zero- or few-shot, for three kinds of local judgments: how plausible each statement is on its own, which pairs of statements contradict each other, and which statements support which answer. A convex soft-logic program then chooses, all at once, how much to trust each statement and which answer to give. Statements that clash with the rest of the evidence are down-weighted even when each looks fine in isolation. The central bet is that hallucinated statements are more often inconsistent with the model's other outputs than with themselves. That bet is testable, and the proposal is built around testing it.

Core Claim

Closed-book knowledge prompting fails mainly because intermediate statements are accepted independently. Enforcing relational consistency among them with soft penalties, rather than hard filters, removes a meaningful share of those failures. This should beat both independent per-statement verification and answer-level majority voting at the same number of model calls, and it needs no weight updates and no retrieval.

The method comes from the soft-constraint literature. Using Soft Constraints in Joint Inference for Clinical Concept Recognition combined local classifier confidences with soft relational penalties in an integer quadratic program. It reached 86.1 F1 against 83.6 for the local baseline and 85.1 when the same rules were made hard. Adapting Coreference Resolution for Narrative Processing adapted a frozen coreference system to folktales by adding soft constraints in an integer linear program, with no retraining. We copy that recipe and change what fills its slots: the "local classifier" becomes a prompted frozen language model, and the "labels" become truth values for the model's own claims. For the solver we use Hinge-Loss Markov Random Fields and Probabilistic Soft Logic. Probabilistic soft logic lets each variable take a continuous truth value between 0 and 1, and finding the most likely assignment is a convex problem. On Cora and Citeseer that paper reports inference in 0.28 to 0.70 seconds, so the solver adds negligible cost next to the language model calls.

Research Program

For a question $q$, the model generates $M$ background statements, as in generated knowledge prompting. It is also given $C$ candidate answers, which come from the task for multiple-choice benchmarks or from sampling for open questions. Let $k_i \in [0,1]$ be the trust assigned to statement $i$ and $a_j \in [0,1]$ the belief in answer $j$. Three prompted judgments supply the evidence:

  • $p_i$ is the model's probability that statement $i$ is true. It is read from the probability of "yes" under a verification prompt, and we also try the indirect-query variant of Do Language Models Know When They're Hallucinating References?, which asks about related facts instead of asking directly.
  • $c_{il}$ is the model's probability that statements $i$ and $l$ cannot both be true.
  • $e_{ij}$ is the probability that statement $i$ supports answer $j$.

Inference solves

$$\min_{k,a}\; \sum_i w_1 \big[p_i (1-k_i) + (1-p_i) k_i\big] + \sum_{i<l} w_2\, c_{il} \max(0, k_i + k_l - 1) + \sum_{i,j} w_3\, e_{ij} \max(0, k_i - a_j) - \sum_j w_4\, s_j a_j \quad \text{s.t.} \sum_j a_j = 1.$$

The four terms do the following:

  • The first term keeps each trust value $k_i$ close to its standalone verification score $p_i$.
  • The second term penalizes trusting two statements the model judged contradictory.
  • The third term penalizes trusting a statement without also believing the answer it supports.
  • The last term rewards answers in proportion to $s_j$, the model's direct score for answer $j$ given no knowledge.

Answers are mutually exclusive, which is the only hard constraint. The weights $w_1$ to $w_4$ are the only free parameters. We fit them on at most 100 development questions per benchmark, and we also report a fixed setting of all weights equal to 1 so the gains cannot be attributed to tuning. The chosen answer is the $j$ with the largest $a_j$.

This differs from the nearest existing work in specific ways:

  • KCTS steers token-level tree search with a trained detector that compares the output to a reference document. We work closed-book at the level of whole statements, with no trained detector and no search.
  • Co-opNet reranks complete candidates with task-specific trained discriminators and cannot recover when every candidate is flawed. We score the parts of the reasoning, so one bad statement can be dropped while the rest are kept.
  • NeuPSL and Teaching Machine Comprehension with Compositional Explanations combine soft logic with neural predicates in order to train models. Here the predicates are prompts to a frozen model and nothing is trained.
  • Step-by-Step Mastery uses "soft constraint" to mean stylistic instructions learned offline with preference tuning. That is a different object from ours, and we do not compete with it.

Pairwise contradiction judgments grow quadratically with $M$, which is the main cost. We handle this in two ways. First, we batch the judgments: one prompt shows a statement alongside all the others and asks which of them it conflicts with, so the cost is $M$ calls rather than $M^2$. Second, every comparison with baselines is made at a matched number of model calls and generated tokens.

Baselines

The naive reference point is direct zero-shot answering: the same frozen instruction-tuned open model, such as LLaMA-3-8B-Instruct, answers each question with no generated knowledge. Every method uses this same model, prompt templates, and decoding settings.

The strongest existing method is generated knowledge prompting itself, rerun on our model. We also quote its published figures on the same benchmarks: 72.5% on NumerSense, 73.0% on CSQA2, and 80.3% on QASC, as reported in Generated Knowledge Prompting for Commonsense Reasoning. Those figures used different models, so the rerun at matched $M$ is the comparison that counts. Beating it means higher accuracy at an equal or smaller call budget, with a bootstrap confidence interval that excludes zero, averaged over three generation seeds.

The second strong baseline is self-consistency: sample many answers and take a majority vote, with the number of samples set so its total calls and tokens equal our method's. This tests whether our extra calls are worth more as structure than as additional votes.

The key ablation is independent verify-and-filter. It uses the same $p_i$ scores, keeps statements above a threshold tuned on the same development set, and answers with those statements. This is exactly our objective with the contradiction and support terms removed ($w_2 = w_3 = 0$). The whole proposal rests on beating this ablation. Beating it on only one benchmark would count as a weak result.

A second ablation replaces the soft penalties with hard constraints. A contradicting pair forces one statement to zero trust, and support becomes strict implication. This repeats the soft-versus-hard comparison of Roth et al. in the new setting.

TruthfulQA has no generated-knowledge baseline on Lacuna. Its reference point is the 58% truthful rate of GPT-3-175B with a helpful prompt reported in TruthfulQA, and our zero-shot and self-consistency reruns serve as the fair comparison there.

Why not just…?

Why not just use self-consistency voting? Voting counts answers but never inspects the reasoning behind them. If the model repeatedly produces the same popular misconception, voting amplifies it. The matched-budget comparison above settles this directly. If self-consistency ties our method at equal cost, the structured inference is not earning its complexity, and we would report that.

Why not just verify each statement independently? This is the most serious threat. The published error analysis says hallucinated statements cause the errors; it does not say that contradictions between statements reveal them. If the model's standalone scores $p_i$ already catch most hallucinated statements, the relational terms add nothing. The $w_2 = w_3 = 0$ ablation tests this. As a diagnostic, we also have annotators label a sample of statements as true or false, in the style of the GKP audit that found 83% of statements factually correct. We then measure how much the contradiction judgments improve detection of false statements beyond $p_i$ alone.

Why not just ask the model to reconcile all statements in one reflective prompt? Reflection-style prompting is cheap. However, Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs reports only 62.9% guideline consistency for its best reflection setup, and reliability across repeated runs varying from a Fleiss' kappa of -0.002 to 0.984. We therefore run a single-prompt reconciliation baseline and also report agreement across seeds. The convex solver is deterministic given its inputs, so we expect lower variance across runs, and that is measurable.

Why not just retrieve documents? Retrieval changes the problem to open-book, and noisy passages bring their own errors, as KCTS notes. We deliberately study the closed-book setting. Retrieved passages could later be added as further evidence variables in the same program, but they are out of scope here.

Proposed Experiments

The main experiment runs every method on CSQA2, NumerSense, and QASC at $M \in \{5, 10, 20\}$ with the batched judgments, and plots accuracy against total model calls.

The second experiment targets the inverse-scaling failure on TruthfulQA and is the decisive test of a stated risk. TruthfulQA shows that larger models more confidently reproduce human misconceptions, and misconceptions held consistently may pass every consistency check. We report the truthful-and-informative rate using the GPT-judge evaluator, which that paper reports agrees 90–96% with human raters, across two sizes of the same model family. If the method helps on commonsense benchmarks but not on TruthfulQA, that result defines where the method stops working: consistency constraints can catch random fabrication but not systematic belief errors. It would be a finding, not a failure to hide.

The third experiment tests whether the trust values $k_i$ are a better hallucination signal than the raw scores $p_i$. We use them to judge whether a given answer is hallucinated on the HaluEval QA split, where HaluEval reports 62.59% detection accuracy for ChatGPT.

The decisive checks are three, in order of importance:

  • The method beats independent filtering at matched budget on at least two of the three commonsense benchmarks.
  • The soft version beats the hard version.
  • The trust values move toward the annotated truth more often than the raw $p_i$ do.

If the first check fails, the central idea is wrong and the note's contribution reduces to a careful negative result about self-consistency as a signal for detecting hallucination.