Chain-of-Rank: Enhancing Large Language Models for Domain-Specific RAG in Edge Device

Juntae Lee, Jihwan Bang, Seunghan Yang, Kyuhong Shim, Simyung Chang

TL;DR

Chain-of-Rank (CoR) is a lightweight alternative to Chain-of-Thought (CoT) reasoning designed specifically for domain-specific Retrieval-Augmented Generation (RAG) on edge devices. By training small-scale LLMs to simply output the IDs of relevant documents before answering, CoR drastically reduces computational overhead (using up to 94% fewer reasoning tokens) while achieving state-of-the-art accuracy.

What Problem Does This Paper Address?

Retrieval-Augmented Generation (RAG) is incredibly useful for specialized domains where precision is critical. To make RAG viable on resource-constrained edge devices, researchers often fine-tune small-scale Large Language Models (LLMs) on a specific target domain.

To help these models filter out irrelevant information from retrieved documents, standard practice relies on reasoning techniques like Chain-of-Thought (CoT). However, the authors identify two major flaws with this approach on edge devices:

  1. High Computational Cost: Generating lengthy text explanations is slow and power-hungry for edge hardware.
  2. The "Reasoning Burden": Small LLMs fine-tuned with parameter-efficient methods like LoRA lack the capacity to consistently generate intricate, flawless reasoning. When a small model generates incorrect reasoning, it actively misleads itself, severely degrading the final answer's accuracy.
Figure 1: Performance for domain-specific RAG on HotPotQA dataset on LLaMA3-8B with LoRA adapter. The marginal effect of CoT (59.2% →60.6%) is because of the generated incorrect reasoning which severely degrades the performance.

Methods

The authors propose Chain-of-Rank (CoR), which simplifies the reasoning step. Instead of forcing the LLM to write out a complex paragraph explaining why a document is relevant, CoR tasks the model with a straightforward objective: identify and output the unique IDs of the relevant contexts, followed immediately by the answer.

Figure 2: Illustration of the proposed chain-of-rank for domain-specific RAG. CoR streamlines the reasoning step, which is easier to be learned.

Model Training

The model is trained using standard supervised fine-tuning with LoRA adapters. The prompt structure explicitly separates the reranking goal from the answering goal. The authors optimize the model using a joint loss function that minimizes both the error in selecting the correct document IDs ($R$) and the error in generating the final answer ($y$):

$$\mathcal{L} = - \sum_{i=1} \log p(y_i | x_i, D_i, R_i) - \log p(R_i | x_i, D_i)$$

Where $x_i$ is the query, $D_i$ represents the retrieved contexts, and $R_i$ is the sequence of relevant context IDs.

Datasets

The method was evaluated on two distinct types of datasets to ensure robustness across natural language and coding domains:

  • HotPotQA: An open-domain question-answering dataset based on Wikipedia. The authors used the "distractor dev set," which challenges the model by providing ten contexts where only a subset is relevant.
  • Gorilla API (TensorFlow, HuggingFace, TorchHub): A benchmark designed to evaluate an LLM's ability to generate correct, executable API calls based on documentation.

Key Insights and Ideas

  • Less is More for Small Models: Intricate reasoning can become a hindrance rather than a help when using low-capability LLMs. Shifting the model's focus to a simple classification/ranking task reduces cognitive overhead.
  • Implicit Attention via Explicit Ranking: Forcing the model to output the relevant document IDs acts as an explicit attention mechanism. It primes the model's internal states to focus only on the golden contexts when generating the final answer tokens.
  • Massive Token Efficiency: CoT and Chain-of-Note (CoN) generate lengthy text blocks that consume substantial context windows and compute. CoR achieves better results using an average of just 8 tokens for its "reasoning" phase.

Key Results

The proposed DSF-CoR (Domain-Specific Fine-tuning with Chain-of-Rank) consistently outperformed vanilla models, standard fine-tuning, and CoT-based methods (like RAFT) across all benchmarks.

  • HotPotQA: CoR achieved an F1 score of 64.11% and an Exact Match (EM) score of 49.23%, outperforming RAFT (60.59% F1 / 46.79% EM).
  • Gorilla API: CoR showed massive jumps in Abstract Syntax Tree (AST) accuracy. On TensorFlow, it reached 95.68% accuracy compared to RAFT's 88.98% and the baseline LLaMA3-8B's 32.11%.
  • Efficiency Gains: On HotPotQA, CoR required only 8.00 tokens for reasoning, compared to 90.15 tokens for CoT and 143.18 tokens for CoN.

Limitations and Future Work

The primary limitation acknowledged by the authors is that this work focuses heavily on domain-specific RAG frameworks that leverage task-specific fine-tuning. They did not thoroughly explore how well CoR translates to general, zero-shot, task-agnostic RAG frameworks where the model cannot be fine-tuned beforehand. However, preliminary testing in the appendix suggests that combining CoR with CoT in a zero-shot setting still yields promising synergistic results.

Summary updated

Abstract

Retrieval-augmented generation (RAG) with large language models (LLMs) is especially valuable in specialized domains, where precision is critical. To more specialize the LLMs into a target domain, domain-specific RAG has recently been developed by allowing the LLM to access the target domain early via finetuning. The domain-specific RAG makes more sense in resource-constrained environments like edge devices, as they should perform a specific task (e.g. personalization) reliably using only small-scale LLMs. While the domain-specific RAG is well-aligned with edge devices in this respect, it often relies on widely-used reasoning techniques like chain-of-thought (CoT). The reasoning step is useful to understand the given external knowledge, and yet it is computationally expensive and difficult for small-scale LLMs to learn it. Tackling this, we propose the Chain of Rank (CoR) which shifts the focus from intricate lengthy reasoning to simple ranking of the reliability of input external documents. Then, CoR reduces computational complexity while maintaining high accuracy, making it particularly suited for resource-constrained environments. We attain the state-of-the-art (SOTA) results in benchmarks, and analyze its efficacy.
BibTeX
@inproceedings{lee2025chain,
  author = {Juntae Lee and Jihwan Bang and Seunghan Yang and Kyuhong Shim and Simyung Chang},
  title = {{Chain-of-Rank: Enhancing Large Language Models for Domain-Specific RAG in Edge Device}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.311},
}
Figure