When a language model is fine-tuned to be both helpful and truthful, the two goals usually agree. Sometimes they do not. In those cases, the model can satisfy "be helpful" by producing a confident, detailed answer it does not actually know, which is to say it fabricates. This note proposes treating that failure the way multi-task learning treats task interference. Helpfulness and truthfulness are two tasks sharing one small set of trainable weights. When their training signals point in opposite directions, the compromise the shared weights settle on is a fluent falsehood. We propose to measure where inside the model that collision happens, and then to physically separate the two updates. Truthfulness updates would be allowed to write only into the part of the model's hidden state that does not directly choose output words. Helpfulness updates would be allowed to write only into the part that does.
The split comes from a finding in hallucination detection, not from an arbitrary design choice. HARP decomposes the model's final word-prediction matrix (the "unembedding," which turns a hidden vector into scores over the vocabulary). It finds that the top singular directions, about 95% of the space, carry how to phrase the output. The remaining roughly 5% carries a signal that predicts whether the answer is correct, reaching 92.9% AUROC on TriviaQA with LLaMA. HARP uses this signal only to detect errors after the fact. Our claim is that the same geometry is useful as a training-time wall between objectives. The wall should also address a known side effect of truthfulness training. Truthfulness updates confined to the reasoning directions cannot directly push up the logit of a stock refusal such as "I have no comment." That shortcut is the kind of brevity bias TruthFlow reports in its steered outputs.
Core Claim
The claim has two parts, and each can fail on its own. The first part is diagnostic. On the preference pairs where helpfulness and truthfulness conflict, the per-example gradients of the two objectives inside a shared low-rank adapter have negative cosine similarity. That conflict is concentrated in the unembedding's reasoning directions rather than being spread evenly. The second part is interventional. An adapter whose truthfulness and helpfulness components are forced into complementary subspaces aligned to the unembedding should do two things. It should reduce hallucination more than a loss-level fix, and it should do so with fewer refusals on answerable questions. It should also beat the same separation along random complementary subspaces of equal size, which is the control that tells us whether the unembedding geometry matters or whether any separation would do.
Relation to Prior Work
Three papers motivate the proposal. CAMP shows that about 11% of preference pairs pit helpfulness against truthfulness. It handles them by softening the margin of the preference-training loss (DPO, direct preference optimization) on those pairs. CAMP never looks at where inside the network the objectives interfere, and that internal question is the gap we target. MTL-LoRA shows that a single low-rank adapter (LoRA, a small trainable matrix added to frozen weights) squeezes multiple tasks into an overlapping, blurred representation. Its task-specific transformations lift commonsense-reasoning average accuracy from 77.6 to 82.1. TruthfulQA shows that more scale does not fix imitative falsehoods and can make them worse. That result rules out "wait for bigger models" as the answer.
Two papers supply the method. LoRI freezes the adapter's down-projection as a random matrix and gives each task a sparse mask on the up-projection, which reduces cross-task interference. We reuse its frozen shared down-projection. We replace its random-and-sparse separation with a geometric one, and LoRI's own separation becomes our key control. HARP supplies the subspace. We use it to route parameter updates, whereas HARP uses it to feed a classifier.
Three papers constrain the design. PIP-KAG mutes whole feed-forward layers (layers 20–29 in LLaMA3-8B) to favor context over memory. It acknowledges that this coarse surgery risks general reasoning, which is why we route updates rather than delete computation. MGAP shows in vision-language models that blunt suppression pushes hidden states off the regions they normally occupy and degrades fluency. That warns us to monitor fluency and general capability, not only truthfulness. TruthFlow finds that the "truth direction" varies by query, so one fixed steering vector is too blunt. Our method learns updates inside a subspace instead of adding a fixed vector. It is still a static subspace, however, and TruthFlow's finding is a real risk that the random-subspace control is designed to expose. CoQE proves that softmax attention entangles context-level and sample-level features and proposes dual-space encoders. Its architecture applies only to single-value outputs, so we borrow its framing but not its model.
Research Program
Let $W_u \in \mathbb{R}^{|V| \times d}$ be the unembedding matrix, where $|V|$ is the vocabulary size and $d$ the hidden width. Take its singular value decomposition $W_u = U \Sigma V^\top$. Let $V_k$ be the top $k$ right singular vectors, with $k$ chosen so the complement is about 5% of $d$ as in HARP. Define the semantic projector $P_S = V_k V_k^\top$ and the reasoning projector $P_R = I - P_S$. We attach adapters only to weight matrices that write into the residual stream, namely the attention output projection and the feed-forward down-projection. For each such matrix $W$, the update is
$$ \Delta W = P_R\, B_T A + P_S\, B_H A, $$
where $A$ is a frozen random down-projection shared by both objectives (as in LoRI), $B_T$ is the trainable truthfulness up-projection, and $B_H$ is the trainable helpfulness up-projection. By construction, the column spaces of the two updates are orthogonal. In words, the truthfulness update can only write into the reasoning directions of the residual stream, and the helpfulness update can only write into the phrasing directions.
$B_T$ is trained with DPO on truthfulness-labelled preference pairs, and $B_H$ with DPO on helpfulness-labelled pairs, drawn from the same pool. We will use per-objective preference labels of the kind CAMP builds, re-annotated if its data is unavailable. Every such set will be checked to be disjoint from TruthfulQA's 817 questions.
The approach rests on one assumption: that directions defined by the final unembedding mean something at intermediate layers. This is the premise behind reading intermediate hidden states through the output matrix. It is plausible in late layers and doubtful in early ones. We therefore treat the range of layers that receive routed adapters as a swept variable and report where the effect lives.
The program has three stages. The first is diagnostic. We train a shared LoRA with the two DPO losses summed. For each preference pair, we compute the cosine between the two objectives' gradients and decompose each gradient into its $P_S$ and $P_R$ components, layer by layer. This tests whether CAMP's conflict set coincides with negative-cosine pairs and whether the conflict concentrates in $P_R$. It is publishable on its own even if the intervention fails. The second stage is the routed adapter above. The third checks whether the routed model keeps an internal truthfulness signal after training, by running HARP's detector on its hidden states before and after.
Baselines
The naive reference points are the supervised-fine-tuned base model and standard DPO trained with a single shared LoRA on the same pooled data. CAMP reports that on HaluEval these reach hallucination rates of 5.75% and 4.35% respectively.
The strongest competitor on this exact question is CAMP itself. It targets the same conflicting pairs with a loss-level adaptive margin and reports a 3.20% HaluEval hallucination rate, along with a 3.28% average truthfulness improvement over standard DPO on TruthfulQA. We will rerun CAMP ourselves on the same backbone, the same preference data, and the same adapter rank and trainable-parameter count, with the same hyperparameter-search budget for every method. Numbers are then compared within our runs rather than against CAMP's reported figures. Beating CAMP means two things together. The first is a lower HaluEval hallucination rate and a higher TruthfulQA True×Informative score (GPT-judge), outside seed variance across three seeds. The second is that helpfulness does not drop, as measured by the refusal rate on answerable TriviaQA and NQ questions and by CAMP's GPT-4-judge helpfulness protocol.
Two adapter-architecture baselines test whether plain multi-task separation suffices. MTL-LoRA uses task-specific transformations with the two objectives as tasks. LoRI uses a frozen random $A$ with disjoint sparse masks on the up-projection, one per objective. Both run at matched parameter counts.
As an inference-time reference we include TruthFlow. On TruthfulQA, TruthFlow raises Llama-3-8B's True score from 52.32% to 64.79%. We rerun it on our backbone and report its informativeness alongside truthfulness, since brevity is its known weakness.
Three ablations isolate the key ingredient. The first replaces $P_S/P_R$ with a random orthogonal split of the same dimensions; this is the decisive control. The second swaps the assignment, so truthfulness goes to $P_S$ and helpfulness to $P_R$. The third applies CAMP's margin inside our routed adapter, to test whether the two fixes are complementary.
Why not just…?
Why not just use CAMP's adaptive margin? CAMP down-weights conflicting pairs, but both objectives still write into the same weights. Its gains over DPO are real but modest: 3.20% versus 4.35% hallucination on HaluEval. It is a genuine threat. If our routed adapter matches CAMP, the architectural story adds nothing, and the head-to-head at matched budget settles this. The CAMP-inside-routing ablation tells us whether the two act on different parts of the problem.
Why not just train separate adapters with LoRI-style disjoint masks? This is the most serious alternative, because LoRI already shows that near-orthogonal random subspaces remove most cross-task interference. If random separation performs as well as unembedding-aligned separation, then our geometric claim is wrong and the contribution collapses to "apply LoRI to alignment objectives." The random-split ablation, alongside the LoRI baseline, is designed to show exactly that. We commit to reporting it either way. The specific prediction that distinguishes the two is informativeness: only the aligned split blocks direct truthfulness writes into vocabulary directions, so only it should reduce refusal-style evasions.
Why not just drop the conflicting pairs from training? Filtering out the roughly 11% of conflicting pairs avoids the collision, but it also throws away the examples that teach the model what to do when it does not know. CAMP's analysis suggests these are precisely where fabrication is learned or unlearned. We include filtered DPO as an additional baseline. If filtering matches our method on HaluEval but raises refusals on answerable questions, the conflict data is necessary and routing is how to use it.
Why not just steer at inference time with TruthFlow? Inference-time steering needs paired truthful and hallucinated hidden states and an extra network at decoding time, and it produces short evasions on trick questions. Our method adds no decoding cost after training. Whether it beats TruthFlow on truthfulness is an open empirical question. The comparison on True×Informative, rather than True alone, is where we expect to win or lose.
Proposed Experiments
We will use Llama-3-8B and Mistral-7B, matching TruthFlow's backbones. The evaluation measures truthfulness, helpfulness, and general capability. Truthfulness is measured by TruthfulQA True, Informative, and True×Informative with GPT-judge, and by the HaluEval hallucination rate on general queries and QA. Helpfulness is measured by the refusal rate on answerable TriviaQA and NQ Open questions and by CAMP's judge protocol. General capability is measured with the eight-task commonsense suite from MTL-LoRA, to catch the kind of reasoning damage PIP-KAG warns about.
Three results would decide the proposal. First, the diagnostic must show that gradient conflict on the conflict set is concentrated in $P_R$ in late layers. If conflict is spread evenly, the premise fails and we stop at the analysis. Second, the aligned split must beat the random split of equal size on True×Informative and on refusal rate. This is the single result the whole proposal rests on. Third, the swapped assignment should do worse than both, showing that the direction of routing matters and not just its presence. A secondary check is that HARP's detector AUROC on the routed model's hidden states stays at or above its level on the base model. That would indicate truthfulness training strengthened, rather than overwrote, the internal correctness signal.
References
- HARP: Hallucination Detection via Reasoning Subspace Projection
- TruthFlow: Truthful LLM Generation via Representation Flow Correction
- Conflict-Aware Adaptive Alignment for LLM Hallucination Mitigation
- MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
- ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
- Mitigating Manifold Departure: Uncertainty-Aware Subspace Rectification for Trustworthy MLLM Decoding
- Reconciling In-Context and In-Weight Learning via Dual Representation Space Encoding