Calibrating the Answer Model Before Asking the Next Question

Language-model agents that play twenty questions, interview suspects, or elicit a user's preferences now choose their questions by expected information gain: the expected drop in uncertainty about the hidden target once the answer arrives. Computing it requires the agent to predict how the other party will answer each candidate question under each candidate hypothesis. That prediction, the answer model $P(\text{answer} \mid \text{hypothesis}, \text{question})$, is just the language model's own guess, and language models are systematically overconfident. When the answer model is overconfident, the agent overrates questions whose answers it cannot actually predict, and it prunes hypotheses on answers it should have doubted. The authors of CA-BED say directly that their method is bounded by the quality of these likelihoods and degrades badly on open-ended Detective Cases. Neither they nor anyone else in this literature proposes a fix.

The proposal is to learn, from cheap offline data, how reliable the agent's predicted answer is for each specific question–hypothesis pair. We then fold that reliability into the information-gain computation as a noisy channel. The central bet is that the harm comes from miscalibration that varies across questions. Some questions, such as "is it a mammal?", get answered as predicted. Others, such as "is it bigger than a breadbox?" or "was the suspect nervous?", get answered unpredictably. A single global correction like temperature scaling cannot separate these two kinds, so it cannot fix the ranking of questions. A per-question reliability estimate can.

Core Claim

Information-theoretic questioners like BED-LLM and CA-BED score questions with a likelihood the language model asserts but never checks. We claim three things. First, most of the success lost to miscalibration comes from question-dependent answer noise. Second, a small learned reliability estimator, trained on (hypothesis, question, observed answer) triples, recovers most of the gap to an oracle answer model. Third, a tuned global noise floor or temperature does not recover it. The claim fails if a well-tuned global correction matches the per-question estimator, and the experiments are designed so that this outcome would be visible.

Concretely, let $\hat a(h,q)$ be the agent's predicted answer to question $q$ if the hidden target were hypothesis $h$. Let $r(h,q) \in [0,1]$ be the learned probability that the real answerer gives that answer. With $K$ possible answers, the corrected answer model is

$$P(A=a \mid h,q) = r(h,q)\,\mathbb{1}[a=\hat a(h,q)] + \frac{1-r(h,q)}{K-1}\,\mathbb{1}[a\neq \hat a(h,q)],$$

and questions are scored by the usual expected information gain under the current belief $b$ over hypotheses,

$$\text{EIG}(q) = H(b) - \mathbb{E}_{a \sim \sum_h b(h)P(a\mid h,q)}\big[H(b \mid q,a)\big],$$

where $H$ is Shannon entropy and $b \mid q,a$ is the Bayes-updated belief. The same corrected likelihood drives the belief update. Calibration therefore affects both which question gets asked and how hard each answer is trusted. When $r$ is close to 1 everywhere, this reduces to the near-hard pruning that overconfident likelihoods produce in practice. When $r$ is a constant, it reduces to the global-noise baseline.

Research Program

The work has three parts.

The first is diagnostic. We define selection regret as the true information gain of the best available question minus the true information gain of the question the agent picked. "True" here means computed with the empirical answer distribution of the answerer, which we can obtain by querying the simulator, or a separate answerer model, many times per (hypothesis, question) pair. We measure selection regret and the calibration error of $P(A\mid h,q)$ for the BED-LLM and CA-BED pipelines on 20 Questions (COMMON, 111 items), Detective Cases (100 scenarios), and MovieLens-100K preference elicitation. We then decompose the calibration error into a global component, which a single temperature removes, and a residual question-dependent component. If the residual is small, the project's premise is wrong and we will report that.

The second part builds the reliability estimator. Training data is generated without any dialogue-level goal labels. We sample a hypothesis $h$ from the catalog, have the questioner propose questions as it would mid-dialogue, record its predicted answer, and query the answerer about $h$. This sidesteps the oracle-goal dependence of Uncertainty-Aware Clarification in LLM Agents with Information Gain, which needs the ground-truth user goal to compute its reward. Our signal needs only answers about known hypotheses. We are honest that for real human users this still requires a population of people answering questions about known items; with a simulator it is free.

The estimator is a small classifier on the questioner's hidden states. We borrow the method from PROBECAL, which cut tool-use calibration error on TabMWP from 0.690 to 0.035 using about 500 training questions. We also borrow from Large Language Models Must Be Taught to Know What They Don't Know, which found that about 1,000 graded examples give most of the calibration benefit. Those works calibrate the model's confidence in its own answers. We calibrate its prediction of someone else's answer, inside a selection loop. That is a different target, and whether the probe transfers to it is an empirical question.

The third part tests whether the effect is a property of the method or an artifact of matched models. Questioner and answerer are drawn from different model families and sizes, and we add a small human-answered subset of 20 Questions items. When the questioner is also the simulator, its predictions can look calibrated simply because they agree with themselves. That masks exactly the failure we care about.

Baselines

The naive reference point is direct prompting: the model asks whatever question it likes and guesses when it chooses. This is the baseline BED-LLM beat by an average of 37.4 percentage points in 20 Questions success, and CA-BED beat on Detective Cases with 46.0% versus 27.6%. It anchors how much information-theoretic selection buys before any calibration.

The strongest existing method is BED-LLM with its Rao-Blackwellized estimator and sample-then-filter hypothesis sets, run unmodified. For the long-horizon setting, the strongest existing method is CA-BED with lookahead and soft updates, also unmodified. For a fair comparison, our method is implemented as a drop-in replacement of the likelihood inside each pipeline. Everything else is held fixed: the same hypothesis sampler, candidate-question generator, turn budget, questioner model, and number of forward passes per turn. Beating these baselines means a higher success rate at equal turn budget, beyond seed variance across the 111 COMMON items and 100 Detective scenarios. The file gives no exact base success rates for BED-LLM itself, only its 37.4-point margin over prompting; we will report both pipelines' absolute numbers from our own reruns. For MovieLens there is no reported result found on Lacuna for a numerical recommendation-quality metric, so that comparison is purely internal.

The ablation that isolates the key ingredient is the global correction. We replace $r(h,q)$ with a single constant noise rate, and separately apply one temperature to the language model's answer probabilities. Both are tuned on exactly the same offline triples used to train the estimator, with the same tuning budget. If the per-question estimator does not beat this, the question-dependence claim is false.

Two further reference points complete the comparison. A self-ensemble likelihood averages answer predictions over several sampled prompts or dropout-style perturbations, in the spirit of CE-BST. There, a ten-model bootstrap ensemble with label smoothing reached the lowest joint-goal calibration error (0.841) and raised top-3 joint goal accuracy on MultiWOZ 2.1 from 69.97% to 84.08%. We give it the same compute budget measured in forward passes. Finally, an oracle answer model using the simulator's empirical answer distributions gives the ceiling. Our target is to close most of the gap between uncorrected BED-LLM and that oracle.

Why not just…?

Why not just apply temperature scaling or a fixed noise floor to the likelihoods? This is the real threat, and it is the main ablation, not something to argue around. A global correction does help with update collapse: it stops one surprising answer from zeroing out the true hypothesis. It cannot, however, change which question ranks first when the problem is that some questions have unreliable answers and others do not. The diagnostic decomposition in part one measures how much of the calibration error is question-dependent. If that share is small, we expect global correction to match us and will conclude the simpler fix suffices.

Why not just ask the model how confident it is in each predicted answer? Verbalized confidence is brittle and sensitive to phrasing. Large Language Models Must Be Taught to Know What They Don't Know reports open-ended calibration error around 30% for such methods, against 10% after light fine-tuning. We include verbalized per-question reliability as a variant of $r(h,q)$, so the comparison is direct rather than assumed.

Why not just use multi-step lookahead, as CA-BED does? Lookahead multiplies the answer model's errors over every simulated branch, so it should make miscalibration worse, not cure it. That matches CA-BED's own report that it degrades on Detective Cases. We run CA-BED with and without the corrected likelihood. Our prediction is that the gain from calibration is larger with lookahead than without.

Why not just train the questioner with reinforcement learning on an information-gain reward? The τ-Bench clarifier of Deng et al. does this and gains 3.7% task success. Its reward requires the ground-truth goal, and the reliability problem then just moves into the reward. Our offline triples need no goal labels and can supply a reliability-corrected reward to such training. We leave that combination as follow-up rather than claiming it here.

Proposed Experiments

The decisive experiments run on 20 Questions COMMON and Detective Cases, with MovieLens-100K as a secondary check. On each, we report success rate at a fixed turn budget, mean turns to a correct guess, selection regret, and calibration error of the answer model. We report these for direct prompting, unmodified BED-LLM and CA-BED, the global-temperature and global-noise ablations, self-ensembles, our per-question estimator, and the oracle.

Three outcomes would settle the question. The first is the size of the question-dependent share of calibration error. The second is whether our estimator closes substantially more of the gap to the oracle than the tuned global correction does, at equal tuning data and forward passes. The third is whether these orderings survive when questioner and answerer come from different model families and on the human-answered subset. We also plot success against the number of offline training triples, from about 100 to several thousand, to check whether the roughly 1,000-example regime reported for confidence calibration carries over.

As a transfer test, we feed the corrected reliability into the parameter-level question selection of SAGE-Agent on ClarifyBench. SAGE-Agent reports 59.73% coverage on ambiguous tasks with 1.39 questions per task. If per-question reliability matters there too, it should raise coverage without raising the question count.