Hop Registers: Giving Language Models a Clean Slot for Intermediate Answers in Multi-Hop Retrieval

Many questions need two or more lookups: to answer "Who is the spouse of the director of film X?", a system must first find the director and then search again for that person. Standard retrieval-augmented generation fetches documents once, using only the original question, and fails when the second document can only be found after the first answer is known. The fixes that work, such as having the model write out a reasoning step and search again, are slow. A faster route reads the model's internal state directly. Layer-wise RAG (L-RAG) shows that the middle layers of a language model already hold the intermediate "bridge" entity before any text is produced. It trains a small network that maps one of those hidden vectors into a retriever's embedding space, which lets it fetch the second-hop document without writing anything out.

This note asks where in the token sequence that intermediate answer actually lives, and whether it gets mixed up with a known numerical oddity: a few tokens whose hidden vectors have far larger magnitude than the rest. In vision transformers these high-norm tokens are background patches that the model repurposes as scratch memory. Giving the model a few extra learnable "register" tokens moves that behavior off the image patches (Vision Transformers Need Registers). We propose the text analogue for multi-hop retrieval. We append a few learnable hop registers after each retrieved passage, fine-tune lightly, and read the next-hop query from the registers' middle-layer states. The bet is that a dedicated slot gives a cleaner and more reliable retrieval signal than a single position that may be dominated by outlier dimensions. We also bet that pulling scratch computation off passage tokens reduces the distraction that comes from stacking many documents in context.

Core Claim

In a retrieval loop that reads hidden states, the quality of the next-hop query depends on which token is read, and not only on which layer. Current systems read one fixed position and ignore how magnitude and information are distributed across tokens. We claim two things. First, in multi-document contexts a few tokens develop very large norms in the middle layers, and their outlier dimensions distort any readout that includes them. Second, explicit hop registers take over this scratch role and give a readout that retrieves the second-hop document more accurately than L-RAG's readout, at essentially the same latency. The first claim is diagnostic and can come out either way; the second is the method.

What Prior Work Establishes and Leaves Open

Three strands motivate the proposal. In vision, Vision Transformers Need Registers found that high-norm patch tokens lose their local content but predict global labels well. It also found that about four added registers absorb this role for under 2% extra compute. SINDER explains the explosion as a power-iteration-like effect of the weight matrices. Outlier Suppression shows that LayerNorm scaling amplifies such outliers in language models specifically. Together these tell us the phenomenon exists in text models and has a mechanical cause, which is why we expect it to affect a hidden-state readout.

On the retrieval side, L-RAG supplies the method we build on: a middle-layer readout projected into a Contriever-style space. It treats that hidden state as a uniform semantic vector and never inspects token-level norms. Our proposal differs only at that point, so L-RAG is the natural base.

Two papers constrain the design. To Sink or Not to Sink shows that high-norm visual tokens in vision-language models carry useful global summaries, and that suppressing them hurts reasoning. So we do not delete or clip high-norm tokens by default. We give their function a home and then test empirically whether the original high-norm tokens carry the bridge entity. Back Attention diagnoses a "hopping too late" failure, in which the first hop resolves too high in the network to leave room for the second. Its fix, letting lower layers attend to higher layers, costs about 1.8x compute per token. Registers are a far cheaper intervention, although we do not claim they solve the closed-book version of that problem.

The closest cross-over work sits in vision-language models. Attention Hijackers shows instruction tokens latching onto high-norm image outliers through the key-value cache, and it names extension to RAG as open. Driving on Registers uses registers as deliberate compression sinks. Neither studies text retrieval chains or reads retrieval queries from registers.

Research Program

The first stage is diagnostic. We take an open 8B instruction model (LLaMA-3.1-8B, to match numbers reported in the file) and run it on 2WikiMultiHopQA, HotpotQA and MuSiQue. The context holds the question, the gold first-hop passage and a varying number of distractor passages. At every layer we record per-token hidden-state norms and flag tokens whose norm exceeds the sequence median by a large factor. We then train linear probes to recover the bridge entity, defined as the first-hop answer embedded by the retriever, from four kinds of positions: high-norm tokens, the final token that L-RAG uses, the bridge-entity mention span itself, and a mean over the passage. This tells us three things: whether high-norm tokens behave like the vision case (they lose local identity and gain global content) or like fixed position sinks with no content, whether their outlier dimensions dominate mean-pooled vectors, and at which layer bridge information peaks for each position type.

The second stage is the method. After each retrieved passage we insert $k$ learnable register embeddings, with $k$ swept over 1, 4 and 8. We fine-tune with low-rank adapters on the training splits using two losses. The first is the usual answer loss. The second is a retrieval loss: a small projection of the registers' layer-$\ell$ state, with $\ell$ chosen from stage one, must score the gold second-hop passage above negatives in the frozen retriever's space. At inference the model reads passage one once, takes the register readout at layer $\ell$, retrieves passage two, and answers. This is a single forward pass per hop and never generates an intermediate query as text. For three- and four-hop MuSiQue questions the procedure repeats, with new registers after each newly retrieved passage.

The third stage checks the mechanism. We ask whether, as in vision transformers, the high-norm spikes move onto the registers after fine-tuning. We ask whether passage-token features become more local, measured by probing each token for its own identity. Finally, we measure whether answer accuracy degrades more slowly as distractors are added. That last test follows Cho et al., who found that reader accuracy falls as more candidate documents are added.

Baselines

The naive reference is single-shot RAG with the same retriever and model: retrieve on the question alone, then read. On 2WikiMultihopQA with LLaMA3.1-8B, MergePRAG reports standard RAG at about 40.51 F1, which anchors the floor. We rerun it in our own setup rather than rely on that figure.

The strongest direct competitor is L-RAG. It solves the same problem, fast second-hop retrieval from hidden states, and our method differs only in where the readout comes from. L-RAG reports 0.352 accuracy on MuSiQue, the best among the methods it evaluated, and about 2.8 seconds per query on 2WikiMultiHopQA against more than 20 seconds for IterRetGen. We reimplement it on the same backbone, retriever, corpus and training data. Its projection network gets the same tuning budget as ours, and its layer choice is swept just as ours is. To isolate the readout from the extra training, we also fine-tune the L-RAG variant with the same low-rank adapters and retrieval loss but without registers. Beating L-RAG means higher second-hop recall@5 and end-to-end F1 on all three datasets, outside seed variance, with latency within 10%.

The accuracy ceiling is the class of iterative text loops, IRCoT (Trivedi et al.) and ITER-RETGEN (Gong et al.). IRCoT reports gains of 11–21 recall points and up to 15 F1 over one-shot retrieval, but no isolated per-dataset number appears in the file. We run it with the same model and retriever. We do not expect to beat it on accuracy. We expect to close most of the gap at a fraction of the latency. MergePRAG's reported 79.34 F1 on 2WikiMultihopQA is a reference from a different regime, since it trains a hypernetwork and writes passages into weights. We report it for context rather than as a matched comparison.

The ablations isolate the key ingredient. We append $k$ untrained random tokens. We add registers but read from the final token. We fine-tune the registers but give them no retrieval loss. We sweep the number of registers.

Why not just…?

Why not just let a learned attention-pooling head read from all positions? This is the most serious threat. A pooling head trained with the retrieval loss could learn on its own to ignore outlier tokens and focus on the bridge-entity span, which would make registers unnecessary. We include it as a baseline under the same budget. If it matches hop registers on recall and F1, we will report that the benefit comes from readout flexibility and not from dedicated slots, and the diagnostic study stands as the contribution. What registers could still add is the stage-three effect on distraction, which a readout head cannot produce because it does not change the passage representations.

Why not just clip or standardize the outlier dimensions in L-RAG's vector? This is cheap and directly targets the second half of Gap 2. However, To Sink or Not to Sink warns that high-norm tokens can carry real summary content, and uniform suppression hurt reasoning in its setting. We run per-dimension standardization and clipping on L-RAG's readout as a baseline. If it recovers most of the gain, the problem was numerical scale, and registers are overkill for retrieval.

Why not just use IRCoT and accept the latency? For offline use that is reasonable. But L-RAG's measurement of more than 20 seconds against about 2.8 seconds per query shows how large the cost is. Growing context also invites the error cascades noted in Wang et al. (2025). Our comparison reports the accuracy–latency frontier explicitly, so readers can see what each seconds budget buys.

Proposed Experiments

The decisive checks fall in order. First, the stage-one probes must show that bridge information is unevenly distributed across positions. If the final token already carries it best and outlier dimensions contribute nothing, the premise fails and we stop at a negative diagnostic result. Second, on 2WikiMultiHopQA, HotpotQA and MuSiQue, hop registers must beat the fine-tuned L-RAG variant, the attention-pooling head and the clipped readout on second-hop recall@5 and answer F1, at matched latency. Third, the gain should widen on three- and four-hop MuSiQue questions, where each misread query compounds. It should also widen as distractor passages grow from 2 to 10, the regime in which scratch computation on passage tokens should be most harmful. Fourth, norm maps after fine-tuning should show the high-norm behavior moved onto the registers. If accuracy improves without that move, the improvement is not explained by the proposed mechanism, and we will say so.