Assistants that try to read between the lines of a request can invent things about the user, much as they invent facts about the world. A user mentions a dinner party, and the model decides the guests are vegetarian. A user asks for a coffee shop, and the model assumes they need quiet. Or the user leaves out a required argument for a tool, and the model fills in a plausible value. Each of these is a hallucination about the user rather than the world, and it is invisible to fact-checkers because no external document can refute it.
We propose to make every inferred preference explicit as a short natural-language definition, such as "the user avoids all animal products, including dairy," and to check each one against the conversation with a trained entailment matcher before it is allowed to shape the response. Each candidate preference ends up stated, inferable from evidence, contradicted, or unsupported. Supported preferences are applied, contradicted ones are dropped, and unsupported ones that matter for the current request trigger a question instead of a guess. The bet is that this check cuts invented preferences sharply without losing the gains of inferring implicit ones.
Core Claim
Preference-inference methods currently judge whether a model recovers hidden preferences. They do not judge whether it fabricates preferences the evidence does not support. Critique-Driven Reasoning Alignment, from Aligning Deep Implicit Preferences by Learning to Reason Defensively, trains a policy with step-by-step textual critiques and reaches 76.3% deep alignment accuracy on DeepPref. Its authors name the cost themselves: pushing models to infer deeply invites over-interpretation and ungrounded assumptions. This "creativity vs. safety" trade-off is the gap that motivates our work.
Our claim is that the fix comes from semantic label embedding, not from more preference training. The label-embedding literature shows that replacing opaque class IDs with definitions and running binary matching ("does this text fit this definition?") generalizes to unseen labels. Label Definitions Improve Semantic Role Labeling pairs sentences with PropBank role definitions and reports +4.4 F1 on unseen predicates. That paper supplies our matching recipe. We treat each preference as a label defined in words, so the matcher can score preferences it never saw in training.
The prediction is specific. Compared with a model that writes its own preference summary and conditions on it, verification should leave recall of real implicit preferences roughly unchanged. It should reduce the rate at which the model asserts preferences whose evidence has been deleted.
Research Program
The system has three parts.
First, a proposer LLM reads the dialogue and the current request. It writes candidate constraints relevant to the request, each as a one-sentence definition. Where two candidates are easily confused, it also writes a contrastive definition, for example "vegetarian, dairy acceptable" versus "vegan." This split borrows from StructCBM in Adaptive Concept Discovery for Interpretable Few-Shot Text Classification, which describes each class with typical-example concepts plus contrastive concepts for confusable pairs and reaches 85.45% 10-shot accuracy on AGNews. StructCBM warns that contrastive concepts grow quadratically with the number of classes. We avoid that by generating them only for pairs the proposer flags as confusable within a single request.
Second, a cross-encoder matcher scores each pair of a dialogue turn and a definition as entailed, contradicted, or neutral. Its output for a preference aggregates over turns. Running the matcher on every turn is slow; definition matching made inference about 3.1× slower in the semantic role labeling work. A dual-encoder prefilter therefore retrieves the top few turns per definition first. This mirrors StructCBM's prune-then-rerank structure.
We train the matcher on PrefEval's training split (Do LLMs Recognize Your Preferences?). Positives are preference-revealing turns. Hard negatives come from deleting or minimally editing those turns. The deletion idea is borrowed from the "utterance removal" construction of Can Tool-augmented Large Language Models be Aware of Incomplete Conditions? and applied to user preferences instead of tool arguments.
Third, a decision rule turns matcher scores into actions. We calibrate the threshold on held-out dialogues so that the rate of applying an unsupported preference stays below a chosen level $\alpha$. This is a much simpler use of split calibration than the full conformal answer sets in TRAC. The generator then answers conditioned only on preferences that pass. When a request-critical slot remains unsupported, it asks a clarifying question.
The analysis question is whether the definition format itself matters. Would the same matcher do as well with the preference given as a bare tag ("diet:vegan")? Do the contrastive definitions specifically reduce confusions between neighboring preferences?
Baselines
The naive reference point is the base model answering zero-shot with the full dialogue in context. PrefEval reports that zero-shot preference-following accuracy falls from roughly 80% to below 10% after ten turns for most models, with o1-preview holding near 50%. The two other naive references from the same paper are a reminder prompt and retrieval of the top five past turns. Both degrade as context grows. All three run on the same backbone, the same dialogues, and the same depths.
The most important baseline is the key ablation: the same proposer writes a preference profile and the generator conditions on all of it with no verification. This isolates the claim. If our system does not beat this ablation on fabricated-preference rate at equal recall, the matcher contributes nothing.
A second ablation replaces the trained matcher with the backbone LLM verifying its own profile through a prompt. We give it the same tokens of dialogue and a comparable inference budget.
The strongest existing preference method is CDRA, which reports 76.3% deep alignment accuracy on DeepPref, compared with 75.0% for supervised fine-tuning and 70.3% for standard GRPO. We will reproduce it on our backbone if the released recipe allows. Otherwise we compare against the reported figure on the same split and say so. Following Zhao et al.'s finding that it generalizes from 10-turn to 70-turn contexts, we also include supervised fine-tuning on PrefEval. Its training data and number of gradient steps will match the total we spend on proposer plus matcher.
For missing tool arguments, the strongest baseline is the Structured Verification checklist. It raises Gemma-3-4B from 20.90% to 68.57% F1 on ToolBench API Replacement in the ICT benchmark, against a human F1 of about 81%.
The success criteria are as follows. On PrefEval, beating the unverified-profile ablation means a clear reduction in fabricated preferences with accuracy within noise of it. On ICT, beating Structured Verification means exceeding 68.57% F1 on Gemma-3-4B.
Why not just…?
Why not just have the model write a preference summary and condition on it? This is the real threat. Explicit profiles alone may recover most of the gain over raw long contexts, because they compress away distractor text. We do not argue around this. The unverified-profile ablation is built into every experiment, and the removal probes described below measure exactly what verification should add. If the fabrication rates are indistinguishable, the proposal's central claim is wrong.
Why not just ask the LLM to double-check its own inferences? HaluEval (HaluEval) shows weak self-recognition: ChatGPT detects hallucinations at only 58.53% accuracy on summarization and 62.59% on QA. HaluEval also finds that showing a correct and a hallucinated answer side by side can worsen discrimination. Both findings suggest that a model auditing its own guesses about a user will be lenient. The self-verification ablation tests whether a small matcher with definitions does better at matched cost.
Why not just use the Structured Verification checklist? It is a free-text prompt that asks whether information is sufficient. It works well for tool arguments, but it was not designed to separate an inferable implicit preference from an invented one across long dialogues. We test it directly on ICT, where it is strongest. We also port it to PrefEval removal probes, where it competes with our system on equal footing.
Why not just fine-tune on PrefEval? Supervised fine-tuning improves long-horizon adherence. However, nothing in its objective penalizes asserting preferences the dialogue never supported. If it already does well on removal probes, it subsumes our method on that benchmark. We would then fall back to the claim that the matcher transfers without retraining to unseen preference types and to ICT.
Proposed Experiments
The decisive experiment uses PrefEval removal probes. For each test dialogue in the implicit choice-based and implicit persona-driven categories, we create a counterpart with the preference-revealing turn deleted and the same distractor context, at 10 and 70 turns. The correct behavior on the counterpart is to not apply the preference, or to ask about it. We report three numbers: preference-following accuracy on the originals, the fabrication rate on the counterparts, and the clarification rate. No reported result was found on Lacuna for fabrication rates under preference removal, so every system, including the baselines, is measured by us.
On DeepPref we report deep alignment accuracy against CDRA's 76.3%. We add a human audit of 200 responses for asserted user assumptions with no textual support, which is where CDRA's over-interpretation risk should show.
For transfer, we apply the matcher without retraining to ICT (ddehun/ICT). Each API parameter's documentation is rewritten as a definition, and an unsupported required parameter means abstain. We compare against Structured Verification's 68.57% F1 on Gemma-3-4B.
Finally, we check that personalization does not add factual hallucination. HaluEval QA questions are preceded by persona-revealing dialogue, and we compare factual hallucination rates for the unverified-profile and verified systems.
The format ablation (definitions versus bare tags) and removing the contrastive definitions are run on PrefEval only. The project succeeds if verification lowers fabrication on removal probes relative to the unverified profile at equal accuracy. It fails if it does not, regardless of the other numbers.
References
- Aligning Deep Implicit Preferences by Learning to Reason Defensively
- Label Definitions Improve Semantic Role Labeling
- Adaptive Concept Discovery for Interpretable Few-Shot Text Classification
- Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
- Can Tool-augmented Large Language Models be Aware of Incomplete Conditions?
- TRAC: Trustworthy Retrieval Augmented Chatbot
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models