Clinicians rarely want a bar chart of feature importances. They want to ask a model pointed questions and get answers they can check. "Among ventilated patients with normal lactate, does this mortality model ever rank a rising creatinine as protective?" "If this patient's urine output had been measured six hours earlier at the same value, would the alert still fire?" In knowledge engineering, questions like these have a name. Competency questions are natural-language requirements that domain experts write before a system is built, then translate into formal queries that serve as a pass/fail test suite. We propose moving this idea from knowledge graphs to clinical prediction models. Clinicians write competency questions. Each question compiles into an executable query against a trained model and its training cohort. Every query returns a checkable answer: a verdict plus the exact patients, model calls, and arithmetic behind it.
The part that is new, and that keeps this from being a nicer interface, is a step borrowed from ontology testing. Before a question is answered, its presuppositions are checked, meaning the facts that must hold for the question to have an answer at all. In the clinical setting, a what-if question presupposes that the hypothetical patient resembles real patients. We turn that presupposition into a data-support test. If the modified patient lies in a region with no nearby real patients, the system reports the question as unanswerable rather than returning a number. Two things follow. Audits become repeatable test suites rather than one-off explanations. And the audit tool itself stops producing the "island" counterfactuals that plague post-hoc explanation methods: hypothetical patients that sit in regions no real patient occupies.
Core Claim
We claim that a clinical audit built from compiled competency questions, with presupposition checks enforced as data-support certificates, will do three things better than current alternatives. It will detect planted behavioral defects in clinical models more reliably than attribution inspection. It will produce fewer unsupported what-if answers than both unconstrained perturbation and a large language model writing free analysis code. And it will do so at a false-alarm rate low enough to use as a regression test. The claim is falsifiable on MIMIC-III and MIMIC-IV using semi-synthetic defects with known ground truth.
Where This Sits in Prior Work
Our requirement side comes from competency-driven knowledge engineering. Building Interoperable Electronic Health Records as Purpose Driven Knowledge Graphs (iTelos) uses competence queries to bound a lean clinical schema, split into shared, domain, and hospital-specific layers. Popularity Driven Data Integration shows that such queries stop engineers from mapping data nobody will ask about. A unified approach to publish semantic annotations of agricultural documents as knowledge graphs translates competency questions into SPARQL, a query language for knowledge graphs, to verify a graph automatically. All three test data structures, never predictive models. We reuse their discipline of writing the questions first and executing them as tests, and we point it at a trained classifier. The iTelos schema gives us a bounded vocabulary of cohort selectors, so questions only reference concepts the application actually needs.
The presupposition mechanism comes from Computing Authoring Tests from Competency Questions: Experimental Validation. That work derives, from each question, the logical conditions that must hold for it to be answerable. It also reports a constraining result. Expert users often rejected the automatically generated tests for negated questions ("which entities do not have X"). We treat negation as a known risk and evaluate it separately, instead of assuming our compiler handles it.
The answer format comes from Interpretability of a Service Robot: Enabling User Questions and Checkable Answers. It parses user questions into a small set of aggregation operations over execution logs (semantic parsing F1 0.84) and returns the calculation trace along with the raw records. That system queried deterministic logs, not a probabilistic model. It had no reason to ask whether a hypothetical input is realistic, because it never built hypothetical inputs.
The support requirement is motivated by The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations. That paper finds that standard counterfactual generators produce explanations disconnected from real data in up to 37% of cases, and it defines a connectedness test we adopt as our main metric. When no nearby real patient exists, our fallback answers what-if questions using matched real patients. That technique comes from Safe and Interpretable Estimation of Optimal Treatment Regimes, which learns a distance metric that weights the variables that drive outcomes and then reasons over matched historical cohorts.
Clinical interpretability systems motivate the gap. Counterfactual and Factual Reasoning over Hypergraphs for Interpretable Clinical Predictions on EHR (CACHE) finds minimal code sets that are sufficient or necessary for a prediction, and experts preferred its explanations 70% of the time. Its explanations are local to one patient, and its code removals are never checked for realism. Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval (KARE) produces free-text reasoning chains that are never verified against anything executable. Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot (AICare) found that senior clinicians engage in "adversarial verification", demanding raw data to stress-test the model, and that ungrounded narrative explanations damage trust. Fixed dashboards cannot serve that behavior; a query language can.
Research Program
The query language. We will build a small typed query language with four kinds of operators:
- Cohort selectors are filters over schema concepts drawn from an iTelos-style, purpose-bounded schema.
- Model probes evaluate the model on a real patient, or on a patient with one variable set to a value.
- Aggregations are counts, means, and quantiles of predictions or deltas, in the style of the service-robot operations.
- Assertions are monotonicity, threshold, invariance, and subgroup-parity checks that turn a question into pass/fail.
Answer semantics. Every compiled query carries its presuppositions. Syntactic ones are handled as in authoring tests; for example, the selected cohort must be non-empty. The central new one is support. Any probe that modifies a patient must yield a point that passes a connectedness certificate: it must be linked by a chain of small steps to real training patients who receive the same predicted class, following Laugel et al.'s justification criterion. The step size is calibrated on held-out real patients.
When the certificate fails, the system does one of two things. It can answer instead over matched real patients who actually have the queried value, using a learned outcome-weighted distance. Or, if too few matches exist, it returns "unanswerable here" and states which presupposition failed.
An answer consists of four parts:
- a verdict;
- the effect estimate;
- the list of patients and model calls used;
- the support certificate.
Compilation. Natural-language questions are translated into the query language by a language model constrained to the query grammar. Unknown phrasings trigger a clarification turn, following the service-robot work. Translation is deliberately kept separate from execution. The execution guarantees then hold regardless of which translator is used, and translation error can be measured on its own.
Test suites. We will elicit competency questions from clinicians for two tasks: in-hospital mortality and 15-day readmission on MIMIC-III and MIMIC-IV, matching KARE's tasks. Further questions will be derived from written guideline statements. Each question is annotated with its intended formal query, which yields a gold set for translation.
Baselines
The naive reference point is standard validation with no auditing: AUROC, precision-recall, and calibration on a held-out split, plus calibration broken down by subgroup. It tells us which planted defects are invisible to the metrics people already report. We run it on exactly the same trained models.
The strongest attribution competitor is automated SHAP and individual-conditional-expectation checks. SHAP (Shapley Additive exPlanations) assigns each feature a share of a prediction. Individual conditional expectation curves trace how one patient's prediction changes as a single feature is swept. Each competency assertion is turned into the nearest check expressible with these tools; for example, the sign of the mean SHAP value within the selected cohort stands in for a monotonicity requirement. This is how attribution is actually used for guideline alignment, as in Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease. It gets the same background data and the same compute. Beating it means higher defect-detection recall at equal or lower false-alarm rate on clean models.
The strongest end-to-end competitor is a frontier language model agent with a code sandbox, the trained model, and the training data. It receives each competency question verbatim and may write any analysis it likes. We give it the same schema documentation, the same number of tool calls per question as our system consumes, and several prompt variants selected on a development split. We also run CACHE-style sufficiency and necessity subsets as a local-explanation competitor for the per-patient questions.
For the core question, which is audit detection of planted defects, we found no reported result on Lacuna for any of these methods. The nearest reported numbers are on other axes. Laugel et al. report up to 37% unjustified explanations from existing counterfactual generators. Perera and Veloso report parsing F1 of 0.84 on their robot-log corpus. CACHE reports a 70% expert preference over attention. Each of these figures comes from a different setting and is not directly comparable to ours.
Two ablations isolate the key ingredient. The first runs our full query language and translator but removes the support certificate and the matched-patient fallback. Probes then perturb patients freely, which measures what presupposition checking buys. The second keeps the support certificate but replaces the grammar-constrained translator with the free-code agent. This asks whether the gain comes from support-aware semantics or merely from a restricted query language.
Why not just…?
Why not just inspect SHAP values and dependence plots? Attribution describes which features moved a prediction. It does not state whether a requirement is met on a defined cohort, and it silently evaluates the model at feature combinations that may not exist in real patients. CACHE's authors report that attention and post-hoc importance give misleading explanations relative to causal sufficiency and necessity. Our first experiment directly measures whether automated attribution checks miss defects that are confined to a subgroup or appear only at realistic combinations of values. If SHAP checks match our detection rate, the conclusion is that the query language adds interface value but not auditing power, and we will report that.
Why not just let a language model with a code interpreter answer the questions? This is the most serious threat. A capable agent can write the same cohort filter and perturbation loop we compile. Its failure mode is that nothing forces it to check whether the modified patients are realistic, and nothing forces its reported numbers to trace back to executed calls. AICare documents clinicians' distrust of plausible but ungrounded narratives. The self-verification results in Self-Verification Improves Few-Shot Clinical Information Extraction (MIMIC-IV ICD-10 F1 from 0.487 to 0.533 only after adding grounding stages) suggest that unverified single-pass outputs are unreliable. Both are indirect evidence. The second ablation settles the question. If the agent, handed our support checker as a tool, matches the full system, then our contribution reduces to the support semantics and the test-suite format, and we will frame it that way.
Why not just use plausibility-constrained counterfactual generators? These are counterfactual methods that add a density or data-manifold penalty. They make individual explanations more realistic, but they still return one synthetic patient per query. They do not return a verdict on a population-level requirement, and they never decline to answer. Our second experiment includes a density-penalized generator variant, scored with Laugel's connectedness test, so it is judged under the same unjustified-answer criterion as our system.
Why not just train an interpretable model? Rule sets become unauditable quickly: 1,447 rules before pruning on ADNI clinical data in PECLIDES Neuro. And deployed clinical models are increasingly high-capacity. The audit is model-agnostic, so we include a gradient-boosted tree and a sparse logistic model among the audited models. Doing so shows that the same competency suite also exposes defects in "interpretable" models.
Proposed Experiments
Planted-defect detection. We train a gradient-boosted tree, a sparse logistic model, and a CACHE-style hypergraph model on MIMIC-IV mortality and readmission. We create semi-synthetic variants in which labels are altered to plant known defects, including:
- a subgroup where a renal marker's effect is reversed;
- a shortcut through a documentation-timing variable;
- a threshold violation that appears only at physiologically plausible combinations of values.
Clean twins of each model are trained on unaltered labels. Clinicians write the competency suite without knowing which defects were planted. We measure recall on defects, false-alarm rate on clean models, and whether the failing query points to the right cohort. The result that would confirm the claim is higher recall than the SHAP checks at a comparable false-alarm rate.
Unjustified answers. Over all what-if questions in the suite on MIMIC-III and MIMIC-IV, we measure the fraction of answers that rely on patients failing Laugel's connectedness test. We compare the full system, the no-certificate ablation, the free-code agent, and the density-penalized generator. On the semi-synthetic data, where the true effect of each intervention is known, we also measure how often each method's verdict is correct. We additionally report how often the full system declines to answer. A system that gets its low unjustified rate mainly by refusing to answer is not the claimed result.
Translation and negation. On the annotated question set, we measure the exact-match rate of compiled queries, the gain from clarification turns, and accuracy separately on negated questions. Negated questions are where the authoring-test study found expert disagreement.
A small adversarial-verification study. Following AICare's protocol, junior and senior clinicians audit a model that contains a hidden defect, once using attribution dashboards and once using the query system. We record defect discovery and time spent. This study is secondary, and its purpose is to check whether checkable answers serve the adversarial senior users that fixed dashboards fail.
The decisive check is the combination of the first two experiments. The idea fails if support-certified competency audits do not beat both SHAP checks and the free-code agent on defect recall and unjustified-answer rate on the same semi-synthetic MIMIC models.
References
- Building Interoperable Electronic Health Records as Purpose Driven Knowledge Graphs
- Popularity Driven Data Integration
- A unified approach to publish semantic annotations of agricultural documents as knowledge graphs
- Computing Authoring Tests from Competency Questions: Experimental Validation
- Interpretability of a Service Robot: Enabling User Questions and Checkable Answers
- The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
- Safe and Interpretable Estimation of Optimal Treatment Regimes
- Counterfactual and Factual Reasoning over Hypergraphs for Interpretable Clinical Predictions on EHR
- Reasoning-Enhanced Healthcare Predictions with Knowledge Graph Community Retrieval
- Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics
- Assessing the Interpretability of Machine Learning Models in Early Detection of Alzheimer's Disease
- Self-verification improves few-shot clinical information extraction
- PECLIDES Neuro: A Personalisable Clinical Decision Support System for Neurological Diseases