Abstract
Identifying missing variables, such as confounders and mediators, is a fundamental challenge in causal inference. Traditional methods rely heavily on domain expertise to hypothesize these hidden factors. This paper introduces a novel benchmark to evaluate the capacity of Large Language Models (LLMs) to perform causal variable inference—identifying missing nodes in a partial causal graph. Through four progressively difficult tasks, the study demonstrates that top-tier models like GPT-4o can leverage their parametric knowledge to act as "proxy experts," effectively reasoning about causal structures to hypothesize missing links.
Introduction
In causal discovery, the primary focus has historically been on inferring the direction of relationships between a fixed set of variables. However, a more fundamental problem often precedes this: identifying which variables are missing from the model in the first place. Missing confounders can lead to significant bias, while missing mediators obscure the mechanisms of action. This paper shifts the focus to "causal variable inference," testing whether LLMs can automate the hypothesis-generation phase of scientific discovery by filling in the blanks of a partial causal map.
Methodology
The researchers developed a benchmark consisting of four tasks designed to test different aspects of causal reasoning:
- Task 1 (Out-of-Context Identification): Selecting the correct missing variable from a list of unrelated distractors.
- Task 2 (In-Context Identification): Selecting the correct variable from a list of distractors that are valid nodes from the same domain, testing the model's ability to distinguish specific causal roles.
- Task 3 (Open-World Hypothesizing): Generating a ranked list of potential missing variables without predefined options.
- Task 4 (Iterative Hypothesizing): Sequentially proposing a chain of mediators between a treatment and an outcome.
Evaluation was conducted using Semantic Similarity and an LLM-as-Judge approach to assess the plausibility of open-ended responses. The benchmark utilized over 4,000 queries from diverse domains, including synthetic graphs (e.g., BNLearn), expert-curated Alzheimer’s disease models, and complex neuropathic pain graphs.
Results and Key Insights
The study found that LLMs are highly context-aware; their performance improves significantly when provided with the structure of the partial graph, indicating they are reasoning about the web of relationships rather than just retrieving simple associations. GPT-4o and GPT-4 emerged as the strongest performers, achieving near-perfect accuracy on identification tasks.
Interestingly, models were more proficient at identifying mediators and colliders than source or sink nodes, suggesting a particular strength in "filling in the mechanism" of known processes. To ensure these results weren't merely a product of memorization, the authors tested models on graphs that post-date their training data, such as the expert-curated Alzheimer’s graph, where performance remained high. Chain-of-Thought (CoT) prompting was also found to specifically improve performance in identification tasks by helping models think through causal dependencies.
Conclusion
While LLMs show remarkable promise as "co-pilots" in scientific discovery, the authors emphasize that they should complement, not replace, human experts. The current benchmark focuses on relatively small graphs, and real-world application will require scaling to hundreds of variables. Future work aims to integrate these hypothesis generators with automated statistical tools to create an end-to-end causal discovery pipeline.
Summary updated