CARE-RAG - Clinical Assessment and Reasoning in RAG

Deepthi Potluri, Aby M. Mathew, Jeffrey B. DeWitt, Alexander L. Rasgon, Yide Hao, Junyuan Hong, Ying DingarXiv 2025 · DOI · arXiv

Abstract

The CARE-RAG (Clinical Assessment and Reasoning Evaluation for RAG) framework introduces a rigorous benchmark to evaluate whether Large Language Models (LLMs) truly reason with retrieved clinical data or merely rely on internal weights to produce plausible answers. Focusing on Written Exposure Therapy (WET) for PTSD, this study reveals a significant "reasoning gap": models often retrieve correct evidence but fail to adhere to strict clinical protocols. The findings highlight the necessity of evaluating logical entailment alongside accuracy in high-stakes medical applications.

Introduction

Retrieval-Augmented Generation (RAG) is frequently proposed as a solution to LLM hallucinations, under the assumption that access to "gold standard" clinical manuals ensures accurate output. However, in mental health, providing the right evidence is insufficient; models must interpret that evidence through structured protocols.

The authors identify a "reasoning gap" where a model might retrieve the correct passage but provide a clinically unsafe answer due to a failure in protocol logic. This is particularly critical in Written Exposure Therapy (WET), where therapeutic success depends on precise adherence to session steps. CARE-RAG asks: Do LLMs actually use retrieved information, or are they "lucky guessers" relying on pre-existing training data?

Methodology

The CARE-RAG framework evaluates models across three dimensions: context fidelity, reasoning complexity, and inference scoring.

Overview of the CARE-RAG benchmarking pipeline, including the WET corpus, retrieval process, and multi-faceted evaluation scoring.

Context Fidelity and Reasoning

Models were tested under three conditions:

  1. Right Context: Exact supporting evidence.
  2. Right Context + Noise: Evidence buried among adversarial distractors.
  3. Wrong Context: Misleading passages to test for over-reliance or hallucinations.

Tasks were stratified by reasoning demands (none, light, heavy). Beyond standard Accuracy, the researchers employed an LLM-as-judge to calculate a Fidelity Score, measuring the logical entailment between the model’s reasoning trace and the retrieved context.

Dataset

The study utilizes a newly curated dataset based on official WET manuals and PTSD clinical guidelines. It consists of 99 evaluation instances (Multiple Choice, Yes/No, and Open-Ended questions) vetted by a multi-panel team of psychologists. Unlike generic benchmarks, this dataset requires strict adherence to clinical logic.

Results and Key Insights

The evaluation of 20 state-of-the-art models, including reasoning-focused models like DeepSeek-R1 and Claude 3.5, revealed several critical trends.

Quantitative performance of small, large, and finetuned models across accuracy, similarity, and inference scores.

The "Right Answer, Wrong Reason" Trap

A major finding was that models often achieve high accuracy on Multiple Choice Questions (MCQs) while maintaining low Inference Scores. This indicates that models frequently guess the correct answer based on internal weights without actually utilizing the provided clinical evidence. In the MCQ category, while several models reached 100% accuracy, their Inference Scores often hovered between 0.75 and 0.85, representing a 15-25% "reasoning deficit."

Clinical Nuance and Noise Sensitivity

Models frequently missed "gray area" nuances. For instance, many incorrectly suggested providing feedback on a patient's writing style during the first WET session, contradicting the protocol's focus on the trauma narrative. Furthermore, the introduction of "noisy" distractors significantly degraded performance, highlighting that "needle-in-a-haystack" retrieval remains a hurdle for clinical deployment.

Discussion and Future Work

While CARE-RAG provides a robust framework, it currently faces limitations such as potential LLM-as-judge bias and a sample size limited to WET protocols. Future work aims to expand the clinical scope and explore Agentic Context Engineering (ACE), where AI agents dynamically prioritize essential references from long-form documents.

Ultimately, the study underscores that until reasoning fidelity reaches near-perfection, LLMs in clinical settings require strict human guardrails and reasoning scaffolds to ensure patient safety.

Summary updated

Abstract

Access to the right evidence does not guarantee that large language models (LLMs) will reason with it correctly. This gap between retrieval and reasoning is especially concerning in clinical settings, where outputs must align with structured protocols. We study this gap using Written Exposure Therapy (WET) guidelines as a testbed. In evaluating model responses to curated clinician-vetted questions, we find that errors persist even when authoritative passages are provided. To address this, we propose an evaluation framework that measures accuracy, consistency, and fidelity of reasoning. Our results highlight both the potential and the risks: retrieval-augmented generation (RAG) can constrain outputs, but safe deployment requires assessing reasoning as rigorously as retrieval.
BibTeX
@misc{potluri2025care,
  author = {Deepthi Potluri and Aby M. Mathew and Jeffrey B. DeWitt and Alexander L. Rasgon and Yide Hao and Junyuan Hong and Ying Ding},
  title = {{CARE-RAG - Clinical Assessment and Reasoning in RAG}},
  year = {2025},
  eprint = {2511.15994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.15994},
}
Figure