Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation

Qiaomu Li, Ying XiearXiv 2026 · arXiv

TL;DR

This paper evaluates whether large "System-1" decision models can replace specialized rerankers in Retrieval-Augmented Generation (RAG) pipelines. While these models can achieve high ranking quality, they often struggle with reliability on long inputs and carry significant latency and memory overhead compared to smaller, dedicated rerankers.

What Problem Does This Paper Address?

In a typical RAG system, a retriever pulls a large set of candidate documents, and a reranker narrows them down to the most relevant few for the final language model to process. Dedicated rerankers are fast but specialized. Recently, "System-1" decision models—LLMs designed to provide quick, structured outputs (like scores or selections) rather than long-form text—have been proposed as more flexible alternatives.

The authors address a critical gap: most reranking studies focus solely on accuracy metrics (like nDCG) on standard benchmarks. They ignore the "systems" reality of production environments: reliability (does the model crash on long text?), latency (is it fast enough for real-time use?), and resource consumption (how much GPU memory does it eat?).

Methods

The researchers conducted a two-phase evaluation:

  1. Broad Screen: They tested 21 open-weight models (9 dedicated rerankers and 12 decision models) across 11 different benchmarks to identify promising candidates.
  2. Controlled Comparison: They performed a deep-dive comparison between a 27B parameter decision model (PPLX Decider) and a 0.6B parameter dedicated reranker (Jina Reranker v3.5).

To ensure a fair comparison, they implemented equal-prefix truncation, a technique that allocates a fixed token budget to every candidate document so that one long text doesn't "crowd out" others in the model's context window. They also evaluated "all-request" scores, which penalize a model for failing to return a valid result, rather than just averaging scores for successful requests.

Fig. 1. Evaluating a reranker replacement in a fixed RAG pipeline. For a given query, both the incumbent dedicated reranker and a proposed decision-model policy receive the same retrieved candidate set. We compare the Top-K candidates they return, including failures and fallback behavior, and measure the serving latency of each policy. The retriever is unchanged, which isolates the reranking step. The downstream LLM is shown for context but is not part of the controlled ranking evaluation. A replacement is justified only when reliability, ranking quality and serving requirements are met together.

Datasets

The study utilized a massive suite of 11 benchmarks totaling over 10,000 queries:

  • BRIGHT: A reasoning-intensive benchmark used for the main controlled study.
  • Multi-hop QA: HotpotQA, MuSiQue, 2WikiMultihopQA, and MultiHop-RAG.
  • Domain-Specific: SciFact (science), QASPER (research papers), GaRAGe (grounded QA), EnterpriseRAG-Bench (company docs), FinanceBench (financial filings), and CRAG (web pages).
  • Transfer Check: CQADupStack from the BEIR suite was used for a final "frozen" confirmation test.
Summary of benchmarks used in the study.

Key Insights and Ideas

  • The "Valid-Only" Trap: Many researchers only report scores for requests a model successfully completes. This paper shows that decision models often fail on the longest, most difficult inputs. Ignoring these failures creates a misleadingly high quality score. How counting failed requests changes the quality picture.

  • Input Handling is the Deciding Factor: Without proper truncation, the PPLX Decider failed on 56% of BRIGHT requests. Reliability isn't a given; it's a result of how you manage the model's context limits.

  • Workload Sensitivity: There is no universal winner. PPLX performed significantly better on enterprise documents, while the dedicated Jina reranker won on multi-hop Wikipedia tasks.
  • The "Tuning" Advantage: Before swapping out a reranker for a massive LLM, try tuning the one you have. Simply adjusting the input policy for the Jina reranker made it 4.5 to 7 times faster than the best decision model configurations with minimal quality loss.
Fig. 2. Counting failed requests removes the apparent quality gain. Each bar splits a policy’s all-request difference from Jina into its ranking gain on the requests it completed (orange) and the relevance Jina finds on the requests it failed (blue). Diamonds and right-hand values give the net difference; parentheses give completed requests out of 1,384.

Key Results

  • Reliability: Default PPLX only completed 44.1% of BRIGHT queries. With truncation, it reached >99% reliability.
  • Quality vs. Latency: While PPLX could match or beat Jina's quality in some setups, it was often much slower. A high-quality two-stage PPLX setup was 5.9x slower than the Jina baseline. The trade-off between ranking quality and serving latency.

  • Resource Footprint: The 27B decision model required ~50 GB of VRAM, compared to just ~4.3 GB for the 0.6B dedicated reranker.

  • Production Verdict: Only one of the tested decision model configurations (PPLX at a 4,096-token budget) passed all three checks for reliability, quality, and latency relative to the incumbent.
Fig. 3. Quality against latency across input budgets (BRIGHT; each point is one budget, from 2,048 to 8,192 tokens). The shaded region meets the quality and latency checks; filled markers also meet the reliability check. Single-call PPLX at 4,096 and 6,144 tokens meets the checks, but only 4,096 is near parity with full-text Jina (6,144 is slower); a tuned Jina is faster; two-stage PPLX is better but slower.

Limitations and Future Work

The authors acknowledge that their latency measurements were serial (one request at a time). In high-throughput production systems, batching and concurrent loading might change the performance dynamics. Furthermore, the study focused on a specific pair of models; other decision models or rerankers might yield different trade-offs. Future work could explore how these models perform when fine-tuned specifically for the reranking task rather than using off-the-shelf weights.

Summary updated

Abstract

Retrieval-augmented generation (RAG) systems often use a dedicated reranker to choose which retrieved candidates reach the language model. Reranking improves relevance but adds latency, so teams look for faster alternatives. System-1style decision models make direct, structured predictions instead of generating long responses. Could one provide comparable reranking quality with less inference overhead? We first screen 21 open-weight models on 11 benchmarks, then compare a promising decision model with a dedicated reranker on identical candidate lists. The default decision-model setup fails on many long candidate sets, so scoring only the requests it completes gives a misleading ranking. Limiting candidate text length largely restores reliability and makes some configurations competitive. A frozen evaluation on new retrieval data supports the reliability and ranking results, but confirms the latency requirement for only one of the two selected configurations. Across workloads, which model ranks better changes, and a tuned reranker remains faster and uses far less GPU memory. A decision model can perform competitive reranking, but replacing a reranker requires weighing quality, latency and memory together, and teams should tune the reranker they already run first.
BibTeX
@misc{li2026can,
  author = {Qiaomu Li and Ying Xie},
  title = {{Can System-1-Style Decision Models Replace Dedicated Rerankers in RAG? A Systems Evaluation}},
  year = {2026},
  eprint = {2610.34816},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.34816},
}
Figure