Capturing Semantic Correctness for Causal Reasoning Evaluation via Symbolic Verification

Paul He, Yinya Huang, Mrinmaya Sachan, Zhijing Jin

Abstract

Large language models (LLMs) are increasingly applied to tasks involving causal reasoning. However, current benchmarks often rely on string matching or surface-level metrics that fail to assess whether a model’s output is formally valid under causal semantics. We propose DoVerifier, a symbolic verification framework that checks whether LLM-generated causal expressions are derivable from a given causal graph using rules from do-calculus and probability theory. This allows us to recover correct answers that would otherwise be marked incorrect due to superficial differences. Evaluations on synthetic data and causal QA benchmarks show that \DoVerifier more accurately captures semantic correctness than standard metrics, offering a more rigorous and informative way to evaluate LLMs on causal tasks.
BibTeX
@inproceedings{he2026capturing,
  author = {Paul He and Yinya Huang and Mrinmaya Sachan and Zhijing Jin},
  title = {{Capturing Semantic Correctness for Causal Reasoning Evaluation via Symbolic Verification}},
  booktitle = {EACL (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.56},
}
Figure