Small Encoders Can Rival Large Decoders in Detecting Groundedness

Istabrak Abbes, Gabriele Prato, Quentin Fournier, Fernando Rodriguez, Alaa Boukhary, Adam Elwood, Sarath Chandar

Introduction

Large Language Models (LLMs) have revolutionized information retrieval through Retrieval-Augmented Generation (RAG). However, these systems face two significant hurdles: hallucinations and computational inefficiency. When a retrieved document lacks the necessary information, LLMs often generate plausible-sounding but incorrect answers instead of acknowledging the gap. Furthermore, invoking massive decoder-only models for every query—even irrelevant ones—leads to substantial waste. This paper explores "groundedness detection" as a solution, proposing the use of small, fine-tuned encoder models as efficient gatekeepers.

Problem Statement

The core challenge is determining if a query is answerable based strictly on a provided context before the generation step. Traditional RAG workflows often skip this verification or rely on the same expensive LLM used for generation to perform the check. This research investigates whether lightweight encoders can perform this task with high accuracy and significantly lower costs.

Methodology

The researchers frame groundedness as a binary classification task (Relevant vs. Irrelevant). They compare two primary architectural approaches:

  1. Encoders: Lightweight models such as BERT, RoBERTa, NomicBERT, and ModernBERT were fine-tuned to score (Query, Context) pairs.
  2. Decoders: State-of-the-art LLMs, including Llama-3 variants (1B, 3B, 8B), GPT-4o, and Claude 3.5 V2, were evaluated in zero-shot and fine-tuned settings.

Experimental Results

The study utilized diverse datasets including SQuAD v2.0, NewsQA, TREC-COVID, and Touché to evaluate performance across general QA and domain-specific retrieval.

Table 1: Performance comparison across datasets and computational costs.

As shown in the results table, fine-tuned encoders demonstrate remarkable performance. On the SQuAD v2.0 dataset, RoBERTa-large achieved an accuracy of 90.2%, outperforming the zero-shot performance of Llama-3.1-8B (81.9%) and approaching the level of GPT-4o (95.5%). Crucially, the inference cost (measured in FLOPs) for these encoders is significantly lower than that of large decoders.

Key Insights

Semantic Matching vs. Generation

Encoders are inherently better suited for semantic matching—the act of comparing two text segments for alignment. Decoders, optimized for next-token prediction, lack this specific inductive bias. This makes encoders more efficient "filters" for groundedness.

Prompt Sensitivity in Decoders

One significant finding was the extreme sensitivity of smaller LLMs to prompt variations. While a large model like GPT-4o is relatively stable, models like Llama-3.2-1B show massive fluctuations in accuracy based on the specific instruction used.

Figure 1: Accuracy variance across different instructions for Llama models on the NewsQA dataset.

This inconsistency makes small decoders unreliable for production gatekeeping, where predictable performance is mandatory.

Economic Viability

Fine-tuning a small encoder is highly cost-effective. The authors note that the cost to fine-tune a model like ModernBERT is equivalent to fewer than 5,000 inference queries on an 8B model. In a high-volume production environment, this investment pays for itself almost immediately.

Conclusion

The research advocates for a "two-tier" RAG architecture. By implementing a fine-tuned encoder as an initial groundedness filter, systems can avoid unnecessary calls to expensive LLMs for ungrounded queries. This approach maintains high accuracy—rivaling closed-source giants like GPT-4o—while slashing inference costs and improving system reliability.

Limitations

The study currently focuses on single-document groundedness. Future work is needed to address multi-document synthesis (multi-hop reasoning) and internal contradictions within documents. Additionally, the performance gap may vary in highly specialized domains not covered in the initial training data.

Summary updated

BibTeX
@inproceedings{abbes2025small,
  author = {Istabrak Abbes and Gabriele Prato and Quentin Fournier and Fernando Rodriguez and Alaa Boukhary and Adam Elwood and Sarath Chandar},
  title = {{Small Encoders Can Rival Large Decoders in Detecting Groundedness}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure