TL;DR
This paper introduces NS-CoT (Neuro-Symbolic Chain-of-Thought), a framework that transforms natural-language clinical trial eligibility criteria into traceable patient counts using the OMOP Common Data Model. By separating LLM-driven interpretation from deterministic symbolic logic, the system ensures that feasibility estimates are not just "black-box" numbers, but reviewable clinical objects with clear execution paths.
What Problem Does This Paper Address?
Before a clinical trial begins, researchers must perform a "site feasibility assessment" to estimate how many potentially eligible patients exist at a given hospital. This is harder than it sounds. Eligibility criteria are written in natural language (e.g., "patients with diabetes on insulin therapy"), which is often ambiguous. Does "on insulin" mean a history of use, current use, or use specifically after a diabetes diagnosis?
In the informatics world, the standard is the OMOP Common Data Model (CDM), which provides a unified schema for health records. However, there is a massive "computability gap" between human-readable protocols and machine-executable SQL. While Large Language Models (LLMs) can generate SQL directly (Text-to-SQL), they often hide critical informatics decisions—like concept selection, temporal anchoring, and exclusion logic—inside an opaque query. If the resulting patient count is wrong, researchers have no way to localize the error or understand the underlying assumptions.
Methods
The authors propose a neuro-symbolic architecture that divides labor between a neural "interpreter" and a symbolic "compiler":
- Neural Interpretation (LLM): An LLM (GPT-4.1) decomposes the natural language query into a Typed Intermediate Representation (IR). This IR identifies clinical mentions, domains (Condition, Drug, etc.), roles (anchor vs. outcome), and temporal relations.
- Symbolic Grounding: The system maps clinical mentions to standard OMOP concepts using vector databases (ChromaDB) and the OMOP vocabulary graph. Crucially, it distinguishes between "vocabulary validity" and "local executability"—checking if the concepts actually exist in the specific site's database.
- Patient-Set Algebra: Instead of jumping straight to SQL, the system represents the logic as a series of set operations:
REQUIRE(inclusion),FORBID(exclusion), and temporal predicates (e.g.,bounded_after). - Deterministic SQL Compilation: A symbolic module compiles the grounded IR into a chain of Common Table Expressions (CTEs). This ensures the SQL is a faithful, reproducible realization of the declared semantics.
Datasets
The study evaluates NS-CoT using several environments:
- SYN-SNUH: A synthetic version of the Seoul National University Hospital database.
- CQ-Bench: A newly introduced benchmark of 100 queries designed to stress-test semantic complexity, ranging from simple condition lookups to complex anchor-bound exclusions.
- FOEM: An existing template-derived benchmark used to assess generalizability.
- COVIC: A multi-site feasibility simulation using 195 criterion-site pairs derived from real COVID-19 trial protocols.
Key Insights and Ideas
- Semantic Discretization: The authors argue that moving from language to data is a process of "discretization." You aren't "discovering" a true cohort; you are "committing" to a specific interpretation. NS-CoT makes this commitment explicit.
- Exclusion as Set Difference: In natural language, "no exposure" is a negation. In NS-CoT, it is formalized as a set-difference operation triggered by a positive event within a specific temporal window relative to an anchor.
- The "Executable-Empty" State: A major "aha" moment is handling cases where a drug class is valid in the dictionary but has zero records in the local database. NS-CoT treats this as an explicit state, preventing the system from failing silently or producing misleading counts.
- Traceability over Accuracy: The goal isn't just a correct number; it's a "traceable tuple." The output includes the IR, the grounded concepts, the SQL, and the execution trace, allowing humans to audit the logic.
Key Results
- Superiority in Complexity: On the CQ-Bench, NS-CoT achieved an 83% exact-count match rate, significantly outperforming the FastOMOP baseline (60%). The advantage was most pronounced in complex temporal inclusion (100% vs 14%) and exclusion (100% vs 36%) tasks.
- Generalization: On the FOEM benchmark, both systems performed similarly well (91% for NS-CoT vs 95% for FastOMOP), suggesting that for simple template-like queries, the extra symbolic structure doesn't hurt but isn't as critical.
- Repairability: In the COVIC multi-site simulation, the authors demonstrated that NS-CoT's intermediate artifacts allowed them to identify and fix systematic errors (a "second-pass repair"), improving the strict match rate from 31.3% to 45.6% and achieving a high site-ranking correlation (0.871).
Limitations and Future Work
- Scope: The current implementation is limited to "single-anchor" temporal logic. It cannot yet handle complex "chains" of events (e.g., A then B then C).
- Demographics: The system struggled with strict demographic alignment (e.g., exact age/gender counts), which the authors identify as a priority for future symbolic refinement.
- Unstructured Data: The framework relies on structured OMOP fields and does not yet incorporate information buried in clinical notes or imaging reports.
- Future Directions: The authors suggest extending the framework to richer temporal reasoning and measuring "semantic loss" across different hospital implementations of the OMOP model.
Summary updated