Language models trained to be honest learn, mostly on general-knowledge questions, when to answer and when to admit uncertainty. When the same model meets a specialist domain such as college physics, pharmacology exams or clinical guidelines, it hallucinates more, and it is also less aware that it is hallucinating. This proposal treats the second failure as a domain-shift problem. The model's internal signal for "I probably know this" is learned where correctness labels are cheap, and it degrades where they are expensive.
We propose to borrow a tool from domain adaptation, a classifier-versus-adversary game that makes features look the same across two data distributions. We use it to transfer a correctness detector from general questions to unlabeled expert questions. The adapted detector then labels preference pairs, "answer" versus "abstain", that teach the model to hedge only where it is likely wrong. The test is simple: at equal answer coverage, does the tuned model give fewer wrong expert answers than models tuned with an unadapted detector, sampling-based uncertainty, or an external verifier?
Core Claim
The claim is narrow. The weak link in honest behaviour on specialist questions is the self-knowledge signal, more than the missing knowledge itself. That signal can be repaired without expert answer labels, using only unlabeled expert questions.
The repair works in two steps. First, a small probe reads a frozen model's hidden states and predicts whether the model's own answer is correct. Second, a domain discriminator, trained through gradient reversal, forces the features the probe reads to be indistinguishable between general and expert questions once we condition on the probe's predicted correctness. Gradient reversal means flipping the sign of the discriminator's gradient so that the feature map learns to fool it.
Value alignment enters through what the probe is used for. It turns "be honest about uncertainty" into domain-specific preference data, without asking the model to compare two plausible answers, which is the pairwise setting HaluEval found to hurt discrimination.
The design is deliberately unlike the standard way of applying domain adaptation to language models. Gap analyses point to a discriminator inside the generation loss. That design runs into the known failure that forced invariance throws away domain-specific information, and that information here is the expert knowledge itself. We instead make invariant only a one-dimensional quantity, the probability that the model is right, and leave the generative representation free.
Prior work and where this sits
Evidence for the gap. Three studies show the gap is real and is not fixed by prompting or scale. TheoremQA finds open models below 15% on theorem-driven STEM questions, even when theorem definitions are supplied. SciBench finds chain-of-thought reasoning improves arithmetic while it can erode the validity of physical premises. TruthfulQA shows larger models can be less truthful.
Those benchmarks measure accuracy. We measure something they do not: whether the model knows when it is in that bottom 85%.
Closest existing work. The nearest system is AWARE. It fine-tunes on biomedical instructions while an external black-box model scores $k$ sampled completions for safety and hallucination. The variance of those scores sets how strongly alignment is weighted, and persistent misalignment triggers abstention. AWARE needs an external verifier and $k$ rollouts per update, and it never models the domain shift itself. Our method needs no verifier at training time, and its uncertainty signal costs one forward pass.
Two other works also train honesty without modelling shift. Self-verification learning trains the model to check its own facts. TruthFlow corrects hidden representations toward truthful directions. Both learn honesty signals where training labels exist and implicitly assume these signals transfer. We test that assumption and repair it when it fails.
Source of the method. The adaptation technique is borrowed. Minimax feature alignment comes from Ren et al., and the gradient-reversal layer from Yu et al.. ATOP supplies a precedent for splitting a representation into a shared, adversarially aligned part and a domain-specific part in a language model. It does this only for BERT essay scorers, and gains about 1.9% QWK across topics.
Source of the preference objective. The objective is Bounded-DPO. Standard DPO (direct preference optimization) trains the model directly on chosen and rejected answer pairs. Bounded-DPO caps how hard the rejected response can be pushed down, and its authors show standard DPO otherwise drifts toward out-of-distribution outputs. That drift is exactly the risk when every rejected response is an expert-domain answer.
Constraints on the design. Two results rule out designs. Prompt engineering against AAOS guidelines found that plain supervised fine-tuning sometimes degraded guideline consistency, which argues against simply fine-tuning on domain text. Domain-adaptation work warns that enforcing invariance on full representations discards discriminative target information. That warning is why we align only the probe's input features, and why we condition on correctness.
Research Program
Stage one: transfer the correctness detector. Take a base model $f$; we use Llama-3.2-3B and Qwen2.5-7B, the scales studied by AWARE and BDPO. Collect source questions with gold answers from general factual QA, namely the HaluEval QA split and TruthfulQA. Label each of the model's greedy answers as correct or incorrect. Collect target questions from the expert benchmarks, using questions only and discarding answers for training.
A small feature map $g$, a two-layer network over a mid-layer hidden state, feeds a correctness probe $c$ and a domain discriminator $d$. Here $d$ also receives the probe's soft prediction, so the alignment is conditional rather than marginal. The training objective is
$$\min_{g,c}\max_{d}\;\; \mathbb{E}_{(x,z)\sim S}\,\ell\big(c(g(h_x)),z\big)\;-\;\lambda\,\mathcal{L}_{\text{dom}}\big(d(g(h_x),\,c(g(h_x)))\big).$$
The symbols are as follows. $h_x$ is the frozen model's hidden state for question $x$ and its answer. $z\in\{0,1\}$ is source correctness. $\ell$ is cross-entropy. $\mathcal{L}_{\text{dom}}$ is the discriminator's cross-entropy loss for telling source from target. $\lambda$ trades accuracy against invariance.
Why the conditioning matters. The correctness base rate differs sharply between domains: well under 15% on TheoremQA for open models, against far higher on general QA. Marginal alignment would push the target's predicted error rate toward the source's, which is wrong. We therefore also reweight target samples by an estimated target correctness rate, obtained by expectation–maximization over the probe's outputs. The ablation section isolates this step.
Stage two: turn the detector into honest behaviour. For each target question, the adapted probe scores the model's answer.
- When the probe is confident the answer is wrong, we build a pair that prefers a calibrated abstention or hedge over the answer.
- When it is confident the answer is right, we prefer the answer over abstention, which guards against reflexive refusal.
- Uncertain cases are dropped.
The model is then tuned with Bounded-DPO, using LoRA at a fixed rank. LoRA here means low-rank adapters that update a small number of added parameters. All pairs compare an answer with an abstention, never two competing answers.
Analysis before any tuning. We first measure whether the problem exists. We check how easily a linear discriminator separates general from expert hidden states. We also check how much a source-trained probe loses in AUROC, the area under the ROC curve for separating right from wrong answers, when moved to each target domain. If that drop is under about 0.03, the adaptation step is unnecessary, and the project reduces to the stage-two recipe with an ordinary probe. We would report that outcome.
Baselines
Naive reference points. The untuned base model answers every question, which sets the hallucination rate to beat. A second reference applies abstention by thresholding the source-only probe with no adaptation and no tuning. It shows how much a plain transferred detector already buys.
Strongest existing method. This is AWARE, re-implemented on our domains with the same base model, LoRA rank, target questions and number of gradient steps. Its black-box verifier is the strongest open model we can run. We give it its $k$ rollouts and report its extra compute rather than equalizing it away. Its paper reports roughly 12% hallucination rate on Llama-3.2-3B in biomedical adaptation. That figure comes from Bio-Instruct, not our benchmarks, so we use it only as a sanity anchor. Beating AWARE means a lower wrong-answer rate at matched coverage on at least two of the three expert benchmarks, at a fraction of its training-time inference.
Sampling-based uncertainty. This baseline replaces the probe with cross-examination consistency in the style of Agrawal et al., which found hallucinated references vary heavily under indirect re-querying. We use agreement across eight samples to build the same answer-versus-abstain pairs, tuned with Bounded-DPO on identical data.
Self-verification and TruthFlow. Both run with their published recipes on the same source data. They test whether honesty training on general data transfers on its own.
Ablations of our key ingredients. The central ablation is identical stage-two training with the unadapted probe, which isolates adversarial adaptation. Further ablations swap conditional for marginal alignment, drop the base-rate reweighting, and swap Bounded-DPO for standard DPO. All methods share a tuning budget of twenty hyperparameter configurations, selected on a held-out labeled target split of 100 questions per domain. That split is never used for training.
Why not just…?
Why not just use self-consistency sampling as the uncertainty signal? This is the most serious threat. It needs no training and is known to track fabrication. Its weakness on expert questions is that a model lacking a theorem can be consistently wrong: TheoremQA's analysis shows errors come from failing to apply rules, not from random guessing. Agreement then signals confidence, not correctness. Our experiment pits the two detectors directly against each other on target AUROC, and also tests their combination. If sampling matches the adapted probe, we report that. In that case the remaining contribution is the single-pass cost and the honesty-pair recipe.
Why not just label a few hundred expert answers and train the probe on them? Where gold answers exist this is cheaper than adversarial training, so we measure it honestly. We plot target AUROC against the number of labeled target examples, from 0 to 1,000, and find the point where supervised calibration overtakes unsupervised adaptation. The method is worth having if that point lies above what is realistic for proprietary or clinical domains. For such domains, the medical comprehension study notes that 44% of failures involve knowledge absent from both text and knowledge graph, so expert labels are slow to obtain.
Why not just rely on AWARE's external verifier? AWARE moves the hard problem into another model, which faces the same expert-domain blind spots. Its gains were shown on biomedical instructions and poisoning, not on theorem-level science. The head-to-head comparison above settles whether a stronger outside judge beats an adapted internal signal. We will also report the verifier's own correctness AUROC on our targets.
Why not just retrieve domain documents? Retrieval attacks the knowledge gap, not the self-knowledge gap. The medical study attributes 32% of errors to retrieval noise, and a model with noisy context still needs to know when to abstain. We treat retrieval as complementary. A secondary run adds it to both our method and the baselines.
Proposed Experiments
Data. Target domains are TheoremQA (800 questions), the SciBench textbook set (869 problems), and NLPEC pharmacist-exam questions, which require Qwen for Chinese. The 34 AAOS osteoarthritis recommendations serve as a small clinical stress test. Each target is split into unlabeled training questions, a 100-question validation split and a labeled test split.
Metrics. The primary metric is wrong-answer rate at matched coverage, the fraction of questions answered. We also report the area under the risk–coverage curve, and target AUROC of each uncertainty signal.
Reference figures. No reported result found on Lacuna for selective prediction or abstention on TheoremQA, SciBench or NLPEC, so all such numbers will be produced here. For context we cite raw accuracy ceilings from the file: under 15% for open models on TheoremQA, 43.22% for GPT-4 with chain-of-thought and Python on SciBench, and 61.8% for KMQA on NLPEC. On the AAOS guidelines, GPT-4 with Reflection-of-Thoughts prompting reaches 62.9% consistency.
Guardrails. Answered-question accuracy must not fall, which guards against a model that "wins" by refusing the easy questions. We check general truthfulness on TruthfulQA, using GPT-judge, which agreed 90–96% with humans, and safety on HEX-PHI. We also check the over-penalization effect: BDPO reported 27.15 on IFEval versus 25.25 for DPO on Qwen2.5-0.5B, and we test whether that effect reproduces under our abstention pairs.
Decisive checks. There are three, and the paper's claim stands only if all three hold:
- A source-only probe loses meaningful AUROC on expert domains, establishing the problem.
- Conditional adversarial adaptation recovers most of that loss without target labels and beats sampling consistency, establishing the mechanism.
- Tuning on adapted-probe pairs lowers expert wrong-answer rates at matched coverage relative to the unadapted-probe ablation and to AWARE, while keeping TruthfulQA and HEX-PHI within one point, establishing the payoff.
References
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- TheoremQA: A Theorem-driven Question Answering Dataset
- SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Alignment-Aware Model Adaptation via Feedback-Guided Optimization
- Do I Really Know? Learning Factual Self-Verification for Hallucination Reduction
- TruthFlow: Truthful LLM Generation via Representation Flow Correction
- Adversarial Domain Adaptation for Classification of Prostate Histopathology Whole-Slide Images
- Fan-Net: Fourier-Based Adaptive Normalization for Cross-Domain Stroke Lesion Segmentation
- Adversarial Topic-Aware Prompt-Tuning for Cross-Topic Automated Essay Scoring
- Rethinking DPO: The Role of Rejected Responses in Preference Misalignment
- Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
- Do Language Models Know When They're Hallucinating References?
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text