TL;DR
Glance is a novel approach to Retrieval-Augmented Generation (RAG) that eliminates the need for a separate reranking model by equipping the generator itself with lightweight adapters. By using a shared prefill for all candidate passages and passage-specific "probes," Glance achieves state-of-the-art reranking quality with minimal memory overhead and improved downstream answer accuracy.
What Problem Does This Paper Address?
Standard RAG pipelines are often bloated and inefficient. They typically rely on three distinct stages: a retriever finds candidates, a reranker (usually a cross-encoder) scores them, and a generator (LLM) produces the final answer.
This setup has two major drawbacks:
- Resource Inefficiency: Serving two large models (reranker + generator) consumes significant GPU memory.
- Redundant Computation: The reranker and generator often process the same text independently. Furthermore, cross-encoders must process each query-passage pair separately, which doesn't scale well as the number of candidates increases.
The authors ask a simple but powerful question: Since the generator is going to read the passages anyway, can it judge all of them using computation shared across the passages?
Methods
The core of Glance is the Decision Adapter. Instead of a separate model, the authors add a lightweight rank-16 LoRA adapter and a 256-dimensional "pointer head" to a frozen 9B parameter backbone (MiMo-V2.6, a hybrid Qwen-based model).
The process works as follows (see Figure 1):
- Shared Prefill: All candidate passages are prefilled into the model's KV cache once.
- Multi-Probe Packing: The model then evaluates short, passage-specific "probes" (e.g., "Does Passage $i$ support the query?") that branch off from this shared cache.
- Scoring: The pointer head turns the hidden states of these probes into relevance scores.
- Unified Serving: Because the reranker is just an adapter, the system can switch between "reranking mode" and "generation mode" in milliseconds without reloading weights.
The authors also introduce Glance-Activated, a variant where the adapter is only active during the probe tokens. This allows the model to prefill passages using its base weights and then continue generating the answer directly from that same cache, avoiding the need to re-prefill the selected context.
Datasets
The authors evaluated Glance across several benchmarks:
- Training: Fine-tuned for just 39 minutes on MS MARCO (4,000 queries).
- Retrieval Evaluation: Tested on BEIR datasets (SciFact and NFCorpus).
- Downstream QA: Evaluated on HotpotQA to see how reranking quality affects final answer accuracy.
- General Skills: Tested on JevBench to ensure the adapters didn't cause the model to "forget" how to handle general logic and decision tasks.
Key Insights and Ideas
- Contextual Awareness: In the
packed_allformat, every probe "sees" all other passages in the shared prefill. This allows the model to compare passages against each other, a capability that standard pointwise cross-encoders lack. - Efficiency through Sharing: By sharing the prefill, Glance is 3.6x faster than making sequential calls to the same 9B model for each passage.
- The "Generator Advantage": Interestingly, Glance often selects context that the generator finds more "usable," even if a dedicated cross-encoder retrieves more "gold" (perfect) paragraphs. This suggests that having the generator pick its own evidence leads to better end-to-end performance.
- Scaling Quality: Unlike zero-shot models that degrade as context grows, a Glance adapter trained on longer contexts actually sees its performance improve as it scores more candidates (up to 200).
Key Results
- Superior Reranking: After brief fine-tuning, Glance achieved an nDCG@10 of 0.741 on SciFact, outperforming the popular
bge-reranker-v2-m3(0.721). - Better Answers: On HotpotQA, using Glance to select the top-3 passages improved the final answer F1 score by 0.045 compared to using a cross-encoder under the same generator.
- Minimal Overhead: Each new "decision skill" (like a new type of reranking) adds only about 0.1 GiB of memory.
- Scaling: As shown in Figure 2, the "trained-long" version of Glance continues to improve in quality up to 200 candidates, significantly gapping the cross-encoder baseline.
Limitations and Future Work
- Latency vs. Cross-Encoders: While much faster than sequential LLM calls, Glance is still slower than small, dedicated cross-encoders (hundreds of ms vs. tens of ms).
- Memory Growth: The current implementation copies the attention cache for each probe branch, meaning memory usage scales with $k \times \text{context length}$. The authors suggest that using a "block mask" in the future could make this growth linear.
- Scope: The study was limited to one model family (hybrid 9B models) and a few datasets. Future work is needed to see if these benefits hold for much larger models (e.g., 70B+) or different architectures.
- Task Interference: Specializing an adapter for reranking caused a slight drop in performance on general decision benchmarks (JevBench), though the authors argue that unified serving (switching adapters per request) solves this.
Summary updated