Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification

Paul He, Yinya Huang, Mrinmaya Sachan, Zhijing Jin

TL;DR

Large language models (LLMs) are increasingly used for causal reasoning, but current evaluation methods often miss semantically correct answers due to superficial differences in expression formatting. This paper introduces DOVERIFIER, a symbolic verification framework that uses do-calculus and probability rules to check for formal equivalence between LLM-generated causal expressions and ground truth, enabling more accurate and rigorous evaluation.

What Problem Does This Paper Address?

LLMs are being deployed in domains requiring causal reasoning, such as policy and medicine. However, evaluating their causal reasoning capabilities is challenging. Existing benchmarks often rely on simple string matching or surface-level metrics like BLEU or BERTScore. These metrics fail to capture true semantic correctness because they can penalize logically valid but syntactically different expressions, or incorrectly reward superficially similar but semantically incorrect ones. This leads to an incomplete picture of an LLM's causal reasoning abilities, potentially overlooking genuinely correct outputs that don't match a reference verbatim. The core problem is how to reliably assess whether an LLM's causal reasoning output is formally valid under the rules of causal inference, not just how it looks.

Methods

The paper proposes DOVERIFIER, a symbolic verification framework. At its heart, DOVERIFIER treats causal reasoning as a formal derivation problem. Given a causal graph (which can be provided or generated by the LLM) and two causal expressions (a predicted one and a ground truth one), DOVERIFIER attempts to derive the predicted expression from the ground truth (or vice-versa) using a set of established rules from do-calculus and probability theory.

The process involves:

  1. Expression Parsing: Converting natural language or symbolic expressions into a standardized internal representation.
  2. Causal Graph Representation: Parsing the causal graph into a format suitable for rule application.
  3. D-separation Testing: Using graph algorithms to check the conditional independence assumptions required by the do-calculus rules.
  4. Proof Search: Employing a breadth-first search (BFS) algorithm to explore all possible valid rule applications, aiming to find a derivation path between the two expressions.

This symbolic approach ensures that equivalence is determined by formal derivability under the causal graph, not by surface-level string similarity.

Datasets

The paper evaluates DOVERIFIER on two types of data:

  • Synthetic Data: A custom dataset of over 10,000 expression pairs generated by applying do-calculus rules to known causal graphs. This allows for precise control over graph structure, derivation paths, and rule usage to test the verifier's internal consistency.
  • CLadder Benchmark: A benchmark for causal question answering that provides causal questions grounded in known DAGs and corresponding ground-truth causal expressions. This dataset is used to evaluate DOVERIFIER's performance on real LLM outputs.

Key Insights and Ideas

  • Semantic Equivalence vs. Syntactic Similarity: The most crucial insight is that causal correctness is about semantic equivalence, not just string matching. LLMs can produce valid causal reasoning that looks different from a reference answer, and DOVERIFIER is designed to bridge this gap.
  • Do-calculus as the Ground Truth: The paper leverages the formal rules of do-calculus as the definitive arbiter of causal validity, moving beyond heuristic or surface-level metrics.
  • Symbolic Verification for LLM Evaluation: DOVERIFIER demonstrates that formal verification techniques, traditionally used in logic and mathematics, can be effectively adapted to evaluate LLM outputs in complex domains like causal reasoning.
  • Hidden Correctness: A significant portion of LLM outputs that are marked incorrect by standard metrics are actually semantically valid but syntactically different. DOVERIFIER uncovers this "hidden correctness."
  • Feedback for Improvement: Beyond evaluation, DOVERIFIER can provide symbolic feedback to LLMs, guiding them to correct their causal reasoning without needing ground-truth answers, by identifying violations of do-calculus semantics.

Key Results

  • Synthetic Data: DOVERIFIER achieved 100% precision and recall on synthetic data with a depth limit of 5, demonstrating its correctness in recovering known valid derivations.
  • CLadder Benchmark: DOVERIFIER significantly improved accuracy across four LLMs (Llama-3.1-8B, Mistral-7B, Llama-3.1-8B-Instruct, Gemma-7B-it) compared to string match and LLM-as-a-judge metrics. For instance, accuracy for Llama-3.1-8B-Instruct increased from 0.66 (LLM-as-a-judge) to 0.90 (DOVERIFIER).
  • Recovering Missed Answers: The symbolic verifier recovered a substantial fraction of semantically correct but syntactically mismatched outputs, highlighting the limitations of existing metrics.
  • Feedback Effectiveness: Applying DOVERIFIER's feedback loop improved LLM performance on causal reasoning tasks, demonstrating its utility for model self-correction.

Limitations and Future Work

  • Scalability: The BFS search can become computationally expensive for very large or deep causal graphs, leading to a tradeoff between completeness and speed. Future work may explore neural-guided proof search or approximate symbolic methods.
  • Feedback Scope: The current feedback mechanism relies solely on the predicted DAG and expression, not the original natural language question. Incorporating question-aware feedback is a future direction.
  • Complex Regimes: For models that already achieve near-ceiling performance on simple benchmarks, the benefits of symbolic verification might be less apparent. Evaluating DOVERIFIER's performance with increasing graph size and model complexity is an area for future research.
  • Incomplete Knowledge: The system can work with incomplete causal knowledge (e.g., partial DAGs), but further exploration in this area is suggested.

Summary updated

Abstract

Large language models (LLMs) are increasingly applied to tasks involving causal reasoning. However, current benchmarks often rely on string matching or surface-level metrics that fail to assess whether a model’s output is formally valid under causal semantics. We propose DoVerifier, a symbolic verification framework that checks whether LLM-generated causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory. This allows us to recover correct answers that would otherwise be marked incorrect due to superficial differences. Evaluations on synthetic data and causal QA benchmarks show that DoVerifier more accurately captures semantic correctness than standard metrics, offering a more rigorous and informative way to evaluate LLMs on causal tasks.
BibTeX
@inproceedings{he2026uncovering,
  author = {Paul He and Yinya Huang and Mrinmaya Sachan and Zhijing Jin},
  title = {{Uncovering Hidden Correctness in LLM Causal Reasoning via Symbolic Verification}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.56},
}
Figure