TL;DR
LLMs often fail clinical reasoning tasks because they rely on surface-level pattern matching rather than formal medical logic—a phenomenon the authors call "schema collapse." This paper introduces CARENLI, a modular agentic framework that routes clinical queries to specialized reasoning "compartments" (Causal, Compositional, Epistemic, or Risk), improving accuracy on clinical NLI benchmarks by an average of 34 percentage points over standard prompting.
What Problem Does This Paper Address?
Despite their fluency, Large Language Models (LLMs) frequently stumble in clinical settings where the "cost of being wrong" is high. The authors identify a core failure mode: schema collapse. Instead of applying the specific inferential rules required for medical tasks (like distinguishing association from causation or weighing risk severity against frequency), LLMs tend to use a generic, undifferentiated reasoning process.
In clinical Natural Language Inference (NLI)—where a model must decide if a clinical statement is entailed, contradicted, or neutral based on a premise—this leads to "fluent but constraint-violating" reasoning. Standard techniques like Chain-of-Thought (CoT) often fail because they don't provide the necessary structural boundaries to prevent the model from taking logical shortcuts.
Methods
The authors propose CARENLI (Compartmentalised Agentic Reasoning for Clinical NLI), a multi-stage agentic pipeline designed to enforce domain-specific reasoning schemas. The framework consists of four specialized roles:
- Router: Analyzes the premise-statement pair and assigns it to one of four "reasoning families."
- Solver: A family-specific agent that executes a predefined procedure (e.g., checking for control groups in causal claims) and emits an explicit reasoning trace.
- Verifier: Audits the solver's trace for factual grounding in the premise and compliance with the required reasoning schema.
- Refiner: Applies minimal corrections if the verifier identifies a partial failure.
By explicitly binding the inference to a specific schema (e.g., an "evidence hierarchy" for epistemic tasks), CARENLI prevents the model from defaulting to simple association-based heuristics.
Datasets
The study utilizes and extends the CTNLI (Clinical Trial NLI) benchmark.
- The Benchmark: Originally introduced by Jullien et al. (2025), it targets four core clinical reasoning capabilities: Causal Attribution, Compositional Grounding, Epistemic Verification, and Risk State Abstraction.
- The Extension: The authors expanded the dataset to 200 expert-validated instances, evenly distributed across the four reasoning types. These instances are generated from parametric templates to ensure that the underlying logical structure remains invariant while clinical details vary.
Key Insights and Ideas
- The "Routing" Bottleneck: The success of the system is highly dependent on the Router. If a task is misrouted to the wrong reasoning family, accuracy drops from ~71% to ~21%.
- Latent Competence vs. Execution: The results suggest that LLMs often possess the latent medical knowledge required for a task but fail to deploy it correctly without "procedural scaffolding." When forced into a structured schema, models that previously failed near-zero baselines showed dramatic recovery.
- Scale is Not a Panacea: Frontier models (including GPT-5.1 and DeepSeek R1) still exhibit systematic failures under zero-tolerance evaluation when using standard prompting. Increasing model size does not automatically fix "schema collapse."
- Auditable Traces: Because CARENLI uses specialized solvers and verifiers, the resulting reasoning traces are more structured and easier for human clinicians to audit compared to free-form CoT.
Key Results
The authors evaluated CARENLI across four backbone models: GPT-5.1, GPT-4.1, GPT-4o-mini, and DeepSeek R1.
- Accuracy Boost: CARENLI improved mean macro accuracy from 23.1% (direct prompting) to 56.8%, a gain of roughly 34 points.
- Reasoning Type Performance:
- Risk State Abstraction: Achieved the highest accuracy at 92.2%.
- Epistemic Verification: Reached 62.7%.
- Compositional Grounding: Remained the most challenging, though it still saw improvements (reaching ~25.6%).
- Refinement Gains: While the Solver provided the bulk of the gains, the Verifier/Refiner stages added a marginal but critical +0.7% overall, with higher impacts (up to +6.9%) in complex causal attribution tasks.
Limitations and Future Work
- Ecological Validity: The benchmark uses template-generated instances. While this allows for precise diagnostic testing, it may not fully capture the linguistic messiness of real-world Electronic Health Records (EHRs) or clinical notes.
- Compositional Reasoning: Despite improvements, models still struggle with "Compositional Grounding" (reasoning over multiple interacting variables like drug, dose, and patient attributes).
- Small Sample Size: With only 200 items, the statistical power for fine-grained subgroup analysis is limited.
- Future Directions: The authors suggest expanding the set of supported reasoning families and testing the framework on less constrained, real-world clinical datasets.
Summary updated