Literature-Grounded Screening of Bad Controls in Point-and-Click Causal Tools

Tools that make causal inference easy share a quiet failure. A user uploads a table, names a treatment and an outcome, and the tool adjusts for "everything else." If any of those other columns is caused by the treatment, adjusting for it biases the answer. Two kinds of column cause this: a mediator, which lies on the path from treatment to outcome, and a collider, which is caused by both. The size of the damage is known. On semi-synthetic benchmarks, simply removing injected post-treatment variables cut heterogeneous-effect error by about 25% for Do-PFN and by 57–72% for CausalPFN (Causal Foundation Models Perform Better without Post-treatment Variables). The data alone often cannot tell a cause of the treatment from an effect of the treatment, because both can produce the same statistical dependencies. Whether "blood pressure at enrolment" came before or after "starting the drug" is usually written down somewhere in the domain literature.

We propose a tool step that runs before any estimation and is driven by retrieval-augmented generation (RAG), meaning a language model answers by first searching a document collection and quoting what it finds. For each candidate covariate, the step asks the literature one narrow question: does this variable influence the treatment, or is it influenced by it? The answer comes back with cited evidence or an explicit abstention. It is then reconciled with what local statistical tests on the data can identify, and the resulting adjustment set is passed to standard estimators. The point is not to rebuild the whole causal graph. Effect estimation needs only each covariate's position relative to the treatment, so the step costs about 2n retrievals for n covariates instead of the n² pairs used by earlier literature-based graph discovery. Each decision carries an auditable citation the user can check.

Core Claim

Retrieval aimed only at the "before or after the treatment" question, combined with local data-driven discovery, should lower effect-estimation error compared with three alternatives: data-only bad-control filtering, ungrounded language-model ordering, and full literature-built graphs. We expect the advantage to appear specifically on domains the language model has seen little of, and to come at a fraction of the cost of pairwise retrieval. If the advantage disappears on rare domains, the remaining contribution is auditability alone. We will report that outcome plainly, not hide it.

Where this sits

The closest systems each stop one step short. CausalAgent is a conversational multi-agent pipeline that runs causal-learn on uploaded data. It uses retrieval only after the analysis, to ground textbook explanations in its reports, so retrieval never touches the analysis itself. EpiCausalX retrieves epidemiology abstracts from seven databases, scores causal claims with Hill's criteria, draws a graph and reads off an adjustment set. It never estimates an effect on data and never checks the literature-built graph against the data. We reuse its idea of weighting evidence by temporality, but apply it per covariate and test it numerically. LACR uses literature retrieval as an oracle for constraint-based discovery and reaches orientation accuracy of 1.0 on ASIA and Sachs once the skeleton is right. This motivates our design, since it shows retrieved text can orient arrows. It also constrains our design: its O(n²) pairwise querying is what we avoid.

Two papers show that the full graph is unnecessary. Causal Inference Using LLM-Guided Discovery proves that a topological order is enough to choose a valid adjustment set, and elicits that order with three-variable prompts. It does so from the model's memory alone, and its authors note this fails for variables rare in the training corpus. CausalGraphBench finds that language-model graph discovery collapses as graphs grow: normalized structural Hamming distance is 0.65 for small graphs and near 1.0 at 50 nodes, and fine-tuning barely helps (0.64). We read these two results together. Language models are poor at whole graphs, and effect estimation does not need whole graphs. That reading justifies the narrow, treatment-centred question.

On the data side, the treatment-centric local discovery filter of Causal Foundation Models Perform Better without Post-treatment Variables supplies the statistical half of our method. It also supplies the gap we target: cases where data cannot separate a parent of the treatment from a child. From RAG research, we borrow the counterfactual "causal discrimination" score of CF-RAG, which penalizes passages that support both a claim and its reverse. In our setting that becomes the rule "evidence that Z causes T and evidence that T causes Z cancel out." The graph-indexed QA systems CausalRAG and CausalRAG2 are relevant only as possible retrievers. They answer text questions and do not feed estimators. Finally, Causal clarity in statistical software argues that tools should state their assumptions rather than print bare estimates. Our per-covariate cited decisions are a concrete version of that demand for the one assumption, valid adjustment, that users most often get wrong.

Research Program

The screen takes as input a treatment T, an outcome Y, a table, and a short plain-language description of each column. For each covariate Z it issues two retrieval queries. The first asks for evidence on the relation between Z and T, phrased both ways. The second asks the same about Z and Y. A language model reads the top passages and assigns one of four labels: Z precedes or causes T; T causes Z; no relation; insufficient evidence. Each label comes with the quoted span that supports it.

We score the passages with the CF-RAG-style rule. A passage that supports both directions counts as zero, and temporality statements such as "measured at baseline" or "following initiation" receive extra weight, following Hill's criteria as used in EpiCausalX. The retrieval index can be a dense or hybrid search over PubMed-scale abstracts. Retriever choice is secondary and will be fixed across all conditions.

Next comes reconciliation with the data. We run treatment-centric local discovery around T. Where the data orient an edge, for example through an unshielded collider, the data decision stands. Where the data leave an edge unoriented, the literature label decides. Where the two openly conflict, the covariate is flagged to the user rather than silently resolved. LACR's finding that a 1988 benchmark graph was outdated relative to modern literature warns that either source can be wrong, so we flag conflicts instead of choosing a winner.

The final adjustment set keeps covariates labelled as pre-treatment and drops descendants of T. It is passed unchanged to TMLE (targeted maximum likelihood estimation), to inverse-probability weighting, and to a causal foundation model, so the screen is a drop-in step for existing tools.

Baselines

The naive reference point is adjusting for every non-treatment, non-outcome column. This is what a point-and-click tool does by default. The ceiling is the oracle adjustment set read from the true graph. All methods are scored by the fraction of the gap between these two that they close, which is the same headroom metric used for the treatment-centric filter.

The strongest data-only competitor is that filter itself, treatment-centric local discovery from Causal Foundation Models Perform Better without Post-treatment Variables. It reports recovering 85.5% to 92.6% of oracle headroom on IHDP and ACIC 2016 with injected post-treatment variables. That is the number to beat on data where the data are informative. Because those injected variables have no real names and no literature, we expect our method to roughly tie it there. Beating it must instead happen on named-variable graphs where the treatment's neighbourhood is statistically unidentifiable. We run it with the same conditional-independence test and significance level used in our reconciliation step.

The strongest knowledge-based competitor is triplet ordering from Causal Inference Using LLM-Guided Discovery, which reports a topological divergence of 1 on the Child graph and a cost of $55 rather than $574 for a 100-node graph. We run it with the same language model and the same token budget as our method, then read the adjustment set from its order.

We add two more knowledge-based comparisons. One is a holistic literature-built graph in the style of EpiCausalX, reimplemented over our index. The other is LACR-style pairwise retrieval over all variable pairs, run on the small graphs where its cost is affordable.

The key ablation removes the retrieved documents and keeps the identical prompts and labels. This isolates retrieval from prompt structure. A second ablation substitutes random passages of matched length, to rule out a context-length effect. A third drops the data-reconciliation step. Beating a baseline means closing more oracle headroom with a paired bootstrap interval excluding zero, across graphs and treatment–outcome pairs.

Why not just…?

Why not just ask the language model, without retrieval, whether each covariate comes before the treatment? This is the real threat. On textbook graphs like ASIA or Cancer, the model's memory will probably match retrieval, and the triplet method already works well from memory. Its own authors concede it cannot handle corpus-rare variables, and that is the regime where we expect retrieval to help. The no-retrieval ablation, stratified into well-known graphs and rare or post-cutoff graphs, settles this. If the gap is near zero on the rare split, we will reframe the contribution as cited auditability, not accuracy.

Why not just use data-driven filtering such as treatment-centric local discovery or PC? These methods have guarantees where the data identify directions, and we keep them for exactly those edges. Their documented weakness is the equivalence-class ambiguity between parents and children of the treatment. OCDB shows that discovery algorithms that look near-perfect on synthetic data degrade unpredictably on real data such as Sachs. The data-only baseline and the no-reconciliation ablation together show how much each source contributes.

Why not just read the timing of each variable from the data dictionary? When measurement timestamps exist and are trustworthy, that rule is simpler and should be used. We will run it as a baseline wherever such metadata are available. Many observational tables are cross-sectional, or record measurement time rather than the time the quantity was determined: a genotype is measured late but fixed at birth. The test is the subset of our tasks where timing metadata are missing or misleading.

Why not just build the full graph from literature, as EpiCausalX or LACR do? Full graphs cost O(n²) queries, and language-model graph quality collapses beyond about 10–50 nodes according to CausalGraphBench. Errors far from the treatment do not matter for adjustment, yet they still cost tokens and can introduce spurious cycles. We compare headroom and cost against both methods on the small graphs where they are affordable.

Proposed Experiments

The main testbed is built from named-variable Bayesian networks with known structure and conditional probabilities: ASIA, Child, Earthquake, Cancer, Alzheimer's, Neuropathic Pain, Covid-19, and the 35 graphs of CausalGraphBench, including its 21 graphs extracted from academic papers. For each graph we sample many treatment–outcome pairs that have at least one descendant of the treatment among the other nodes. We simulate observational data and compute the true average effect by intervening in the network.

To test the rare-knowledge claim fairly, we split graphs into well-known benchmarks and paper-derived graphs whose source papers postdate the language model's training cutoff. We also remove the source paper from the retrieval index, so the method cannot simply look up the answer. Sachs serves as a real-data check against its accepted 16-edge graph.

Metrics are as follows. Role-labelling accuracy, with emphasis on recall of post-treatment variables, since missing one is the costly error. Absolute average-effect error and fraction of oracle headroom under TMLE, weighting and a causal foundation model. Retrieval calls, tokens and dollars per task. Citation faithfulness, measured by a blinded human audit of whether each quoted span actually supports its label.

As a real-world sanity check, we rerun the NHEFS smoking-cessation analysis. Causal clarity in statistical software reports average-effect estimates from 3.318 to 3.448 kg across five estimators with a pre-treatment covariate set. We then append plausible named post-treatment columns that we simulate from the treatment, such as post-cessation caloric intake. The screen should remove them, citing literature, and return estimates inside that range, while all-covariate adjustment should leave it.

Three results would decide the project. The first is whether retrieval beats the no-retrieval ablation on the rare split and not merely on the textbook split. The second is whether the combined method exceeds data-only filtering on graphs where the treatment's neighbourhood is statistically unoriented. The third is whether its cost scales roughly linearly with covariate count while pairwise retrieval scales quadratically. A failure on the first leaves a cheaper, auditable tool. A failure on the second means the literature adds nothing the data did not already give, and the screen should be dropped in favour of the data filter.