CARF: Context-Aware Reasoning Supervision for Scientific Evidence Screening

Nishtha Agarwal, Adam Visokay, Astha Wahi, Kwabena Owusu Aninkora, Amy Gu, Tyler McCormick, Li LiuarXiv 2026 · arXiv

TL;DR

CARF (Context-Aware Reasoning Framework) is a human-in-the-loop framework that adapts Large Language Models (LLMs) for scientific evidence screening by training them on expert-corrected reasoning rather than just final "include/exclude" labels. By teaching models how to think about complex eligibility criteria, CARF achieves near-perfect recall (96.4%) in specialized biomedical domains, effectively acting as a high-precision filter that preserves ambiguous cases for human oversight.

What Problem Does This Paper Address?

Systematic reviews are the backbone of evidence-based medicine, but they are notoriously labor-intensive. Researchers must often screen thousands of titles and abstracts to find a handful of relevant studies. While LLMs seem like a natural fit for automating this "needle in a haystack" problem, they often stumble on nuanced, context-dependent decisions.

Standard automation approaches usually treat screening as a binary classification task (Include vs. Exclude). However, scientific screening isn't always black and white. It requires interpreting complex clinical language and subjective judgments. When an LLM receives only a final label as supervision, it misses the "why" behind an expert's decision—the specific contextual considerations that make a study eligible or ineligible.

Methods

The authors propose CARF, a four-stage framework designed to bridge the gap between model reasoning and expert judgment:

  1. Blind Screening: A "teacher" model (DeepSeek-R1-32B) screens articles and generates a Chain-of-Thought (CoT) reasoning trace along with a decision, without knowing the expert's original label.
  2. Expert Correction: Domain experts review cases where the model’s decision or reasoning diverges from the verified "gold standard." Instead of just fixing the label, the experts provide targeted natural-language feedback to correct the model's flawed reasoning.
  3. Reasoning Dataset Construction: A dataset is built pairing the article text with either the model's original (correct) reasoning or the expert-corrected reasoning.
  4. Model Adaptation: A smaller "student" model (DeepSeek-R1-14B) is fine-tuned on this dataset using Low-Rank Adaptation (LoRA).
Figure 1: Overview of the Context-Aware Reasoning Framework (CARF) applied to the title and abstract screening stage of a systematic review, illustrating four stages: blind model screening, expert correction of disagreements, construction of an expert-reasoned dataset, and reasoning-based model adaptation through LoRA fine-tuning.

By focusing on the process of reasoning, the student model learns the contextual boundaries of the task, such as how to handle borderline cases or specific domain-specific exclusion criteria.

Datasets

The researchers evaluated CARF on a specialized corpus related to Verbal Autopsy (VA) and Cause-of-Death (COD) ascertainment—a complex field in global health where data is often sparse and context-heavy.

  • Corpus: 5,263 screening records exported from DistillerSR, sourced from PubMed, Embase, Scopus, ACM Digital Library, and arXiv.
  • Training Set: 3,861 articles used for expert correction and dataset construction.
  • Test Set: 1,000 articles reserved exclusively for final evaluation.

Key Insights and Ideas

  • The "Self-Correction" Phenomenon: One of the most striking findings was that CARF-trained models developed a self-corrective behavior. During inference, the model would often start by considering evidence for inclusion, but then "re-read" the criteria, identify a contextual violation, and change its mind to exclude the study. This behavior was strongly associated with correct exclusions.
  • Reasoning > Labels: Training on reasoning traces proved far more effective than "label-only" fine-tuning. The reasoning-supervised model was better at identifying eligible studies that simpler models missed.
  • Safe Ambiguity: In systematic reviews, a "False Negative" (missing a relevant study) is a disaster, while a "False Positive" (including an irrelevant one) is just a minor inconvenience for the next stage of review. CARF is intentionally "sensitive"—it is more likely to include ambiguous or borderline studies, ensuring that human experts still get the final say on the most difficult cases.
  • Feedback Efficiency: The authors found that even a relatively small amount of expert feedback (500 notes) significantly boosted recall compared to using only 100 notes, suggesting that the quality of reasoning supervision is a powerful signal.

Key Results

CARF outperformed both traditional encoder baselines and state-of-the-art zero-shot models in the primary metric of Recall (sensitivity).

Comparison of CARF against baselines, showing superior recall and F1 scores.
  • High Sensitivity: CARF achieved a recall of 0.964, correctly identifying 160 out of 166 eligible studies.
  • Balanced Performance: It maintained the highest F1 score (0.719) among all evaluated methods, including frontier models like Claude 3.5 Sonnet and DeepSeek-V3.
  • Workload Reduction: While CARF had more false positives than the conservative PubMedBERT baseline (119 vs 56), its near-perfect recall makes it a much safer tool for the initial stages of a systematic review.

Limitations and Future Work

  • Domain Specificity: The study focused on a single, highly specialized domain (VA/COD). It remains to be seen if these results generalize to broader biomedical fields or other types of scientific screening.
  • The "Gold Standard" Problem: The framework relies on expert annotations, which are themselves subject to human error and inconsistency.
  • Causal Link of Self-Correction: While the "self-corrective" reasoning behavior is highly correlated with better performance, the authors note that further experiments are needed to prove that this behavior causes the improvement rather than just being a byproduct of adaptation.
  • Cost of Expertise: Collecting high-quality natural language feedback from domain experts is expensive. Future work could explore ways to make this feedback loop more efficient or use LLMs to "critique" each other under lighter human supervision.

Summary updated

Abstract

Systematic reviews synthesize biomedical evidence to inform clinical and public-health decision-making, but require domain experts to manually screen thousands of studies against detailed eligibility criteria. We investigate whether language models can assist with this bottleneck by learning from expert feedback on title and abstract screening decisions and model-generated reasoning. We introduce the Context-Aware Reasoning Framework (CARF), a human-in-the-loop framework that uses expert feedback on model-generated reasoning to adapt language models to context-dependent screening tasks. We evaluate CARF on systematic reviews of verbal autopsy and cause-of-death ascertainment methods used in global public health mortality surveillance. CARF shows competitive screening performance compared to evaluated models while retaining high sensitivity, and is more inclined to retain ambiguous, context-dependent studies for human review. Together, these findings suggest that expert-corrected reasoning is a promising supervision signal for context-dependent scientific screening, while retaining a role for human oversight.
BibTeX
@misc{agarwal2026carf,
  author = {Nishtha Agarwal and Adam Visokay and Astha Wahi and Kwabena Owusu Aninkora and Amy Gu and Tyler McCormick and Li Liu},
  title = {{CARF: Context-Aware Reasoning Supervision for Scientific Evidence Screening}},
  year = {2026},
  eprint = {2610.38041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38041},
}
Figure