Introduction
Causal Structure Learning (CSL) is a fundamental task in data science, aimed at uncovering the underlying directed acyclic graph (DAG) that represents cause-and-effect relationships within observational data. A prevalent approach to CSL is score-based learning, which treats the problem as an optimization task: searching for a graph structure that maximizes a predefined score function. For decades, the Bayesian Information Criterion (BIC) has served as the standard score function due to its mathematical consistency. However, BIC often struggles in real-world scenarios characterized by limited sample sizes, noise, or non-standard data distributions. Manually designing specialized score functions for diverse domains is a labor-intensive process requiring deep statistical expertise.
The L-SFE Framework
To address these challenges, the authors introduce L-SFE (LLM-enhanced Score Function Evolution), a novel framework that leverages the reasoning and coding capabilities of Large Language Models (LLMs) to automate the discovery and refinement of score functions. L-SFE operates through a bi-level optimization process.
The Upper Level involves an evolutionary search within the space of possible score functions. Unlike traditional evolutionary algorithms that rely on random mutations, L-SFE utilizes an LLM (specifically GPT-4o mini) as an intelligent search operator. The LLM is provided with Python implementations of existing score functions and their associated performance metrics. It then generates new candidate functions through three primary operations:
- Mutation: Refining the logic or parameters of a single high-performing function.
- Crossover: Synthesizing ideas from two distinct successful functions.
- Injection: Proposing entirely new mathematical concepts to ensure diversity in the search space.
The Lower Level executes the causal search. For each score function generated by the LLM, a standard search algorithm (such as Greedy Hill Climbing) is employed to find the optimal causal graph for a given training dataset. The "fitness" of the score function is then evaluated based on how closely the resulting graph matches the ground truth, typically measured using Normalized Hamming Distance.
Experimental Evaluation
The effectiveness of L-SFE was rigorously tested across several environments:
- Synthetic Discrete Data: Utilizing Erdos-Renyi and Scale-Free graphs with multinomial distributions.
- Synthetic Continuous Data: Employing various Structural Equation Models (SEMs), including Linear Gaussian, Exponential, and Gumbel distributions.
- Real-World Case Study: Analyzing a COVID-19 pandemic dataset from the UK to evaluate the framework's performance on complex, messy observational data.
Key Insights and Results
A significant finding of this research is the LLM's ability to act as a "statistical intuition engine." Rather than producing arbitrary code, the LLM independently rediscovered sophisticated statistical techniques. For instance, it incorporated Dirichlet priors for smoothing and utilized quadratic penalties to mitigate overfitting in small datasets.
As shown in the evolution plot, the framework consistently improves upon the standard BIC baseline. The LLM-generated functions proved to be more "aggressive" in penalizing model complexity when data is scarce, leading to more accurate structure identification where BIC tends to be too lenient.
Quantitatively, L-SFE achieved the lowest Structural Hamming Distance (SHD) in the majority of tested scenarios—8 out of 10 discrete settings and 7 out of 10 continuous settings. Furthermore, the evolved score functions demonstrated high "cross-algorithm generality," maintaining superior performance even when paired with search frameworks different from those used during their evolution. In the real-world COVID-19 case study, L-SFE significantly improved "arrow accuracy," correctly identifying causal directions that human-designed scores often missed.
Conclusion and Future Work
L-SFE represents a significant step toward automating the "art" of statistical modeling. By evolving Python code, the framework produces interpretable and immediately deployable score functions that outperform traditional metrics. While the current approach involves some computational overhead due to the LLM's involvement, future work aims to optimize hyperparameter tuning within the evolved functions and explore using LLMs to design more efficient search strategies for navigating the vast space of possible causal structures.
Summary updated