Task Vectors as a Cheap Cure for Instruction-Detached Translation

Large language models asked to translate without examples ("zero-shot") often fail in a deceptive way. The output reads fluently but is in the wrong language, leaves the source untranslated, adds sentences that were never there, or loops on a phrase. On a recent stress benchmark, GPT-4o and Claude-3.7-Sonnet hallucinate on about a third of hard inputs, and an open 7B translation model does so on 84% of them. Adding a few example translations to the prompt ("in-context learning") is the standard remedy. The cost is that those example tokens stay in the model's working memory, the key-value (KV) cache, for every generated token. Because decoding is limited by how fast memory can be read, that cost matters.

This proposal tests a specific bet. Much of the benefit of in-context examples for translation may be carried by one compact internal activation, a "task vector" that encodes what task and which target language. We can extract that vector once per language pair, store it in kilobytes, and add it to a bare zero-shot prompt. The idea itself is simple. The contribution is a falsifiable prediction about which hallucinations this fixes. We expect the vector to repair failures of task identity: wrong language and untranslated copies. We expect it to leave failures of source grounding untouched: additions and repetition loops. If that split holds, it tells practitioners exactly when a free, training-free fix suffices and when they still need fine-tuning or re-ranking.

Core Claim

HalloMTBench, from Challenging Multilingual LLMs, sorts LLM translation hallucinations into two families. In instruction detachment, the model ignores the translate-into-X request. In source detachment, the model obeys the request but drifts from the source content. Our claim is that the two families have different causes inside the model, and that task-vector patching separates them. A task vector extracted at an intermediate layer from faithful few-shot translations should recover most of the few-shot reduction in instruction detachment. It should recover little of the reduction in source detachment, if few-shot prompting yields any such reduction at all. The cost is on the order of kilobytes of stored state, against tens of megabytes of KV cache for a few hundred demonstration tokens. That estimate comes from standard 8B architecture arithmetic: roughly 128 KB of keys and values per token with grouped-query attention.

We further predict that repairing source detachment requires a signal that depends on the source sentence itself. A fixed per-language vector cannot supply that, because by construction it is the same for every input.

What Exists and What Is Missing

Two papers supply the method, and one constrains it. In-Context Learning Creates Task Vectors shows that few-shot prompts are compressed into a hidden state at the final token of the demonstrations, around layer 15 in LLaMA-7B. Patching that state into a demonstration-free pass recovers 80–90% of few-shot accuracy, against 10–20% for query-only inference. Its tasks, including French-to-English, are all single-token. One Task Vector is not Enough scales this to 3,096 tasks on Llama-3-8B. It confirms the same "Goldilocks" layer 15, finds that a single vector fails on multi-step generative tasks such as JSON generation, and shows that injecting sequential "sub-task vectors" nearly doubles correct-token probability. Sentence translation is a generative task, so this second paper predicts that a single vector will fall short. Our hypothesis gives that prediction a specific shape: we expect the shortfall to fall mostly on source detachment.

Three papers address translation hallucination from other angles, and each leaves room for this work. Hallucination-focused Preference Optimization fine-tunes ALMA-7B-R on millions of synthesized preference triplets and reduces hallucination by an average of 96% on five seen English-to-X pairs and 89% on unseen ones. It is the strongest existing mitigation, but it requires offline data synthesis and weight updates. Prefix Embeddings for In-context Machine Translation trains static continuous vectors on a frozen GPT-Neo 2.7B and gains about 5 BLEU with one example and 2 BLEU with twenty. It never measures hallucination, and its vectors are learned rather than read off the model's own few-shot computation. CASAL amortizes activation steering into one trained sub-module and cuts factual QA hallucination by 30–40%. It argues that inference-time steering is expensive, but that cost comes from per-query monitoring; our intervention is an unconditional vector addition with no detector.

Two further papers motivate the mechanistic analysis. Model Introspection for NMT hallucinations finds that hallucinating encoder-decoder models show low, local, and static source contribution. The idea of reading source contribution off the model's internals motivates our analysis, but raw attention is a poor stand-in for it: HalOmi reports that raw attention maps are near-random as hallucination detectors, so we use attribution rather than attention.

Research Program

We work with two open decoder-only models. The first is Llama-3-8B-Instruct, where layer 15 is already characterized. The second is Seed-X-PPO-7B, whose 83.99% HalloMTBench rate leaves the most room to move. For each target language, we build many k-shot prompts from faithful parallel sentences, with k from 1 to 8. We run them, and record the residual-stream state at the final separator token at each layer. Averaging over demonstration sets gives one vector per layer and per language pair. A separate variant averages within one domain (TED, biomedical MED, or noisy MTNT), which asks whether domain as well as language is encoded.

We test three injection schemes on the zero-shot prompt. The first replaces the separator-token state once, following Geva et al. The second adds the vector at every decoding step with a tuned scale; this costs one vector add per step. The third is a sub-task schedule after Tikhonov et al. Here we extract states at matched relative positions inside the demonstration targets (start, middle, end) and inject each at the corresponding relative position during generation, using the source length to predict target length.

The analysis has two parts. First, we score every output with HalloMTBench's category-level judging, whose ensemble agrees with human experts at 93.68–100%. This lets us report hallucination rates separately for Untranslated Content, Incorrect Target Language, Extraneous Addition, and Repetition. Second, we measure how much each generated token depends on the source span, using gradient-times-activation attribution over the source tokens. The prediction is that task vectors change the language of the output while leaving the source-contribution profile of detached outputs static.

Baselines

The naive reference point is zero-shot prompting with the benchmark's own translation instruction. It sets the floor that any intervention must beat. It also gives the zero-extra-memory cost point.

The main training-free competitor is k-shot prompting, with k in {1, 2, 4, 8}. The demonstrations are the same ones used to extract the vectors, and their KV states are shared across queries through prefix caching. This is the strongest competitor because it is what practitioners already do, and prefix caching removes its prefill cost. For each k we report hallucination rate by category, COMET on clean text, and decode-time memory per sequence. The task vector "beats" k-shot if it reaches at least 80% of k-shot's reduction in instruction detachment while cutting per-sequence extra memory by two or more orders of magnitude. This mirrors the recovery fraction Geva et al. report for single-token tasks.

A strengthened-instruction baseline isolates the most mundane explanation. It restates the target language by name, writes the instruction in the target language, and forces the first generated token to a target-language tag. If this matches the task vector on instruction detachment, the vector carries nothing beyond language identity.

A trained-vector ablation isolates the key ingredient, namely that the vector is read from the model's own few-shot computation rather than learned. It trains a prefix embedding in the style of Prefix Embeddings at the same position and layer, on 100 bitext pairs, with the same number of free dimensions. A layer sweep and a random-vector control, matched in norm, separate a real task vector from a generic perturbation.

The strongest existing method is Scaled CPO from Tang et al.. It reports an average 96% hallucination reduction on seen English-to-X pairs and 89% on unseen ones, on ALMA-7B-R, measured with BLASER 2.0-QE on NewsCrawl-derived data. No reported result was found on Lacuna for CPO or for k-shot prompting on HalloMTBench. We will therefore run CPO ourselves on Llama-3-8B-Instruct for a subset of pairs. We do not expect the training-free vector to beat it overall. Success means matching it on instruction detachment at a small fraction of its data and compute, and characterizing the source-detachment gap it leaves.

Why not just…?

Why not just use few-shot prompting with prefix caching, since the demonstrations are shared? Prefix caching removes repeated prefill work, but every decoding step still reads the demonstrations' keys and values. Decoding is limited by memory bandwidth, so that read is the dominant cost. This alternative is a real threat when k is small: if one short example already removes most instruction detachment, the savings from a vector are modest. We settle it by plotting hallucination rate against decode-time memory per sequence for all k and all vector variants. The claim survives only if the vector sits clearly on a better cost-quality frontier than 1-shot.

Why not just write a stronger instruction or force the target language? For the Incorrect Target Language category this may work as well, and we treat it as the most likely way the proposal fails. It is less likely to fix Untranslated Content, where the model copies or paraphrases in the source language despite knowing the task, or to carry domain conventions. The strengthened-instruction baseline and the MED-domain vector experiment decide this. If prompt-forcing matches the vector everywhere, we report the task-vector route as unnecessary for translation.

Why not just fine-tune with preference optimization, which already removes 96% of hallucinations? It is stronger, and we say so. However, it needs millions of synthesized triplets and a weight update per model. Our question is how much of that benefit is available with no training at all, and which failure category remains. Running CPO on the same backbone and splitting its gains by category shows whether CPO's advantage is concentrated in source detachment. That result would itself be informative about what fine-tuning buys.

Why not just train a soft prompt or amortize the steering in the style of CASAL? Trained vectors may outperform extracted ones. The trained-vector ablation measures that gap directly at equal parameter count. If a trained prefix wins by a wide margin, the practical recommendation becomes "train 0.0001% of parameters," and the extracted vector's value reduces to being training-free and instant for a new language pair.

Decisive Checks

Five results would decide the proposal. The first test is the category split on HalloMTBench's 11 English-to-X directions. We expect the vector to close at least 80% of the few-shot gap on Untranslated Content and Incorrect Target Language, and substantially less on Extraneous Addition and Repetition. If both families move equally, the mechanistic claim is wrong, even if the method is useful.

The second is a conflict test. Injecting a French vector into a German-instruction prompt should yield French output, as in Geva et al.'s override experiments. This would confirm the vector encodes target language rather than a generic "be careful" direction.

The third is a collateral-damage check. On clean TED, MED, and MTNT sentences in French, German, and Portuguese, COMET must not drop relative to zero-shot. On HalOmi, partial hallucination rates must not rise.

The fourth is the attribution analysis. On outputs that remain source-detached after patching, source-contribution profiles should stay low and static. This would show that the residual failure is a grounding problem that no fixed per-language vector can address. It would also motivate source-dependent steering as the next step.

The fifth checks robustness. We repeat the headline comparison on a 3-bit GPTQ-quantized Llama-3-8B-Instruct. There we test whether compression raises the instruction-detachment rate and whether the kilobyte-scale vector still restores it. This is the regime where memory is scarcest and a few hundred demonstration tokens are most expensive.