Glance: The Generator Is Its Own Reranker

Adilet Metinov, Chingiz ArzievarXiv 2026 · arXiv

TL;DR

Glance is a novel approach to Retrieval-Augmented Generation (RAG) that eliminates the need for a separate reranking model by equipping the generator itself with lightweight adapters. By using a shared prefill for all candidate passages and passage-specific "probes," Glance achieves state-of-the-art reranking quality with minimal memory overhead and improved downstream answer accuracy.

What Problem Does This Paper Address?

Standard RAG pipelines are often bloated and inefficient. They typically rely on three distinct stages: a retriever finds candidates, a reranker (usually a cross-encoder) scores them, and a generator (LLM) produces the final answer.

This setup has two major drawbacks:

  1. Resource Inefficiency: Serving two large models (reranker + generator) consumes significant GPU memory.
  2. Redundant Computation: The reranker and generator often process the same text independently. Furthermore, cross-encoders must process each query-passage pair separately, which doesn't scale well as the number of candidates increases.

The authors ask a simple but powerful question: Since the generator is going to read the passages anyway, can it judge all of them using computation shared across the passages?

Methods

The core of Glance is the Decision Adapter. Instead of a separate model, the authors add a lightweight rank-16 LoRA adapter and a 256-dimensional "pointer head" to a frozen 9B parameter backbone (MiMo-V2.6, a hybrid Qwen-based model).

The process works as follows (see Figure 1):

  • Shared Prefill: All candidate passages are prefilled into the model's KV cache once.
  • Multi-Probe Packing: The model then evaluates short, passage-specific "probes" (e.g., "Does Passage $i$ support the query?") that branch off from this shared cache.
  • Scoring: The pointer head turns the hidden states of these probes into relevance scores.
  • Unified Serving: Because the reranker is just an adapter, the system can switch between "reranking mode" and "generation mode" in milliseconds without reloading weights.
Figure 1: One resident backbone, shared passage encoding, passage-specific decisions. Shaded parts run with the decision adapter (LoRA + pointer head). In the main pipeline the selected passages are re-prefilled and answered with the adapter off; in the activated variant the passages are prefilled with the base weights, so generation continues the same cache, which still holds all k candidates.

The authors also introduce Glance-Activated, a variant where the adapter is only active during the probe tokens. This allows the model to prefill passages using its base weights and then continue generating the answer directly from that same cache, avoiding the need to re-prefill the selected context.

Datasets

The authors evaluated Glance across several benchmarks:

  • Training: Fine-tuned for just 39 minutes on MS MARCO (4,000 queries).
  • Retrieval Evaluation: Tested on BEIR datasets (SciFact and NFCorpus).
  • Downstream QA: Evaluated on HotpotQA to see how reranking quality affects final answer accuracy.
  • General Skills: Tested on JevBench to ensure the adapters didn't cause the model to "forget" how to handle general logic and decision tasks.

Key Insights and Ideas

  • Contextual Awareness: In the packed_all format, every probe "sees" all other passages in the shared prefill. This allows the model to compare passages against each other, a capability that standard pointwise cross-encoders lack.
  • Efficiency through Sharing: By sharing the prefill, Glance is 3.6x faster than making sequential calls to the same 9B model for each passage.
  • The "Generator Advantage": Interestingly, Glance often selects context that the generator finds more "usable," even if a dedicated cross-encoder retrieves more "gold" (perfect) paragraphs. This suggests that having the generator pick its own evidence leads to better end-to-end performance.
  • Scaling Quality: Unlike zero-shot models that degrade as context grows, a Glance adapter trained on longer contexts actually sees its performance improve as it scores more candidates (up to 200).

Key Results

  • Superior Reranking: After brief fine-tuning, Glance achieved an nDCG@10 of 0.741 on SciFact, outperforming the popular bge-reranker-v2-m3 (0.721).
  • Better Answers: On HotpotQA, using Glance to select the top-3 passages improved the final answer F1 score by 0.045 compared to using a cross-encoder under the same generator.
  • Minimal Overhead: Each new "decision skill" (like a new type of reranking) adds only about 0.1 GiB of memory.
  • Scaling: As shown in Figure 2, the "trained-long" version of Glance continues to improve in quality up to 200 candidates, significantly gapping the cross-encoder baseline.
Figure 2: Quality vs. candidate count (data of Table 4). Trained adapters keep improving as more candidates share one prefill (k=200 on the chunked path); the zero-shot adapter degrades beyond k=20. Peak memory grows from 15 GiB (k=10) to 56 GiB (k=100) because of per-branch cache copies.

Limitations and Future Work

  • Latency vs. Cross-Encoders: While much faster than sequential LLM calls, Glance is still slower than small, dedicated cross-encoders (hundreds of ms vs. tens of ms).
  • Memory Growth: The current implementation copies the attention cache for each probe branch, meaning memory usage scales with $k \times \text{context length}$. The authors suggest that using a "block mask" in the future could make this growth linear.
  • Scope: The study was limited to one model family (hybrid 9B models) and a few datasets. Future work is needed to see if these benefits hold for much larger models (e.g., 70B+) or different architectures.
  • Task Interference: Specializing an adapter for reranking caused a slight drop in performance on general decision benchmarks (JevBench), though the authors argue that unified serving (switching adapters per request) solves this.

Summary updated

Abstract

Retrieval-augmented generation commonly serves separate models for reranking and answering, duplicating model residency and passage processing. We introduce Glance, which equips the generator with a lightweight decision adapter and pointer head to score candidate passages from a shared prefill. Short relevance probes branch from the cached context, producing one score per passage without separately prefilling each query-passage pair. A single resident 9B backbone supports generation, general decisions and reranking, with approximately 0.1 GiB of additional memory per decision skill. After 39 minutes of MS MARCO fine-tuning, Glance achieves nDCG@10 of 0.741 on SciFact and 0.345 on NFCorpus, compared with 0.721 and 0.326 for bge-reranker-v2-m3. On 300 HotpotQA questions, its selected context improves answer F1 by 0.045 over cross-encoder-selected context under the same base generator (paired bootstrap 95% CI [0.014, 0.076]). In a scaling study, reranking quality keeps rising up to 200 candidates (43k tokens) scored from one shared prefill, where it exceeds the cross-encoder by 0.058 nDCG@10, and a probe-only activated adapter lets generation continue from the original passage cache. These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.
BibTeX
@misc{metinov2026glance,
  author = {Adilet Metinov and Chingiz Arziev},
  title = {{Glance: The Generator Is Its Own Reranker}},
  year = {2026},
  eprint = {2610.40454},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.40454},
}
Figure