CaDO: Formal Verification-Guided Multi-Agent Framework for Causal Dataset Repair and Generation

Songli Yu, Qi LuoarXiv 2026 · arXiv

TL;DR

CaDO is a multi-agent framework that uses symbolic causal engines to automatically diagnose, repair, and augment causal reasoning datasets. By translating natural language into formal queries, it replaces unreliable LLM "judgments" with verifiable computation, significantly improving the accuracy and reliability of downstream models.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being trained on causal reasoning tasks, such as predicting the effect of an intervention or answering counterfactuals. Because real-world causal data is scarce, researchers often rely on programmatically generated datasets like CLadder or Corr2Cause.

However, these "synthetic" datasets are far from perfect. They frequently contain incorrect labels, flawed rationales, and structural imbalances (like "shortcut cues" where a specific word might accidentally correlate with a "yes" answer). Existing methods to clean this data usually involve "LLM-as-a-judge," which is inherently circular: you are using the very skill you want the model to learn to decide if the training data is correct. This leads to a "garbage in, garbage out" cycle where the judge's own reasoning errors are baked into the repaired dataset.

Methods

The authors propose CaDO (Causal-Engine-Guided Multi-Agent Data Optimization), a framework that shifts the source of truth from LLM opinions to executable formal logic. The process follows three main stages:

  1. Formal Causal Reasoning (The Verification Core): Instead of asking an LLM if an answer is right, a Formalizer Agent translates the natural language story into a formal causal query (a graph, a set of probabilities, and a specific question). This query is then handed to a Causal Engine—a symbolic program that applies the rigorous rules of causal inference to calculate the mathematically certain answer.
  2. Multi-level Diagnosis:
    • Sample-level: A Diagnoser compares the original dataset's label and rationale against the engine's output. To ensure the formalization was correct, Critic Agents perform back-translation and independent checks to provide a "confidence tier" for every proposed repair.
  3. Dataset-level: A Profiler scans the entire dataset to find "coverage gaps" (missing types of questions) and "shortcut cues" (spurious correlations that models might exploit).
  4. Automated Repair and Generation: Based on a rule-based plan, the system applies six operators to fix labels, replace rationales, filter unanswerable items, and debias the text. Crucially, Explorer and Expander Agents generate new, diverse examples to fill gaps, which are then "round-trip" verified by the causal engine before being added.
Fig. 2. Overview of the CaDO framework. CaDO integrates formal causal verification, multi-level diagnosis, and automated dataset repair and generation. Bottom: The verification core uses a label-blind Formalizer Agent to translate natural-language questions into formal causal queries, which are evaluated by a causal engine to produce verified answers, derivations, and verification statuses. Top left: At the sample level, the Diagnoser identifies annotation errors through grounding, rationale consistency, and verification confidence checks, while independent Critic Agents assess the reliability of proposed repairs. Top middle: At the dataset level, the Profiler detects insufficient query-type coverage, label imbalance, and spurious shortcut cues to construct a rule-based repair plan. Top right: Six operators perform targeted correction, filtering, debiasing, and data augmentation. Newly generated examples are validated through a round-trip verification process before inclusion in the repaired dataset. This design ensures that repair and generation decisions are grounded in executable causal semantics rather than LLM judgments.

Datasets

The framework was evaluated on three major causal reasoning benchmarks:

  • CLadder: A dataset covering all three levels of the "Ladder of Causation" (association, intervention, counterfactuals).
  • CauGym: A benchmark focused on multi-variable causal graphs with numerical probability answers.
  • Corr2Cause: A dataset testing the ability to infer causal structures from correlations and independences.

The authors created "corrupted" versions of these datasets by injecting seven types of defects (e.g., flipping labels, reversing causal links, adding shortcut sentences) to rigorously test CaDO's ability to recover the ground truth.

Key Insights and Ideas

  • Verification over Judgment: The most powerful idea is that for formal domains like causality, correctness is computable. By grounding the repair process in a symbolic engine, CaDO avoids the hallucinations and biases of LLM judges.
  • Label-Blind Formalization: The Formalizer Agent never sees the original label. This prevents it from simply agreeing with the dataset and forces it to extract the underlying causal logic purely from the text.
  • Precision Matters: In data cleaning, "doing no harm" is vital. The authors found that LLM judges often "break" perfectly good data (changing correct labels to wrong ones). CaDO's multi-agent critique system ensures that repairs are only made when there is "firm ground."
  • Dataset-Level Health: Repairing individual rows isn't enough. By identifying and removing shortcut cues, CaDO prevents models from learning "cheats" and forces them to learn actual causal reasoning.

Key Results

  • Superior Accuracy: Models fine-tuned on CaDO-repaired data outperformed those trained on the strongest baselines in 16 out of 18 experimental settings, with accuracy gains of up to 9.1%.
  • High Reliability: At least 95% of the samples flagged by CaDO were truly defective. On the CLadder dataset, CaDO modified 92.2% fewer correct labels than a standard LLM-based judge.
  • Robustness to Corruption: As shown in the sensitivity analysis, CaDO's performance remains remarkably stable even as the percentage of defective rows in the training data increases from 10% to 50%.
Fig. 3. Test accuracy of external evaluation models fine-tuned on each method’s output (mean over all seeds, headline test sets). Grey band: from the corrupted to the clean training data (oracle). Loss filter, confident learning and GPT-Agree were run with the Qwen models only. Overlapping markers are shifted within their row. Table III gives the values with significance marks.

Limitations and Future Work

While highly effective, CaDO is currently limited by the scope of its Causal Engine. It can only verify and repair questions that can be expressed in the formal languages the engine understands. If a causal question involves nuances not captured by the current symbolic representation, the system must abstain.

Future work aims to extend the framework to handle more complex, naturally occurring causal defects and to expand the library of formal causal queries the engine can process. The authors also suggest that this "formalization-verification" loop could be applied to other structured domains like law, medicine, or software engineering.

Summary updated

Abstract

High-quality causal reasoning datasets are essential for developing models with reliable causal understanding, particularly in specialized domains. However, existing datasets frequently suffer from incorrect annotations and structural imbalances, which can lead to biased predictions and limited generalization. To address this issue, we propose CaDO (Causal-Engine-Guided Multi-Agent Data Optimization), a multi-agent framework for automated diagnosis and repair of causal reasoning datasets. Our framework consists of three key stages: formal causal reasoning, multi-level diagnosis, and automated data repair and generation. First, a Formalizer Agent translates natural language questions into formal causal queries to minimize semantic deviations and make retrieval and verification more explicit and tractable. Second, a dual-level diagnosis mechanism employs independent Critic Agents to identify sample-level errors while analyzing query-type coverage and label distributions to detect dataset-level structural imbalances. Finally, the framework employs reusable repair operators grounded in formal causal semantics to correct erroneous samples, while Explorer and Expander Agent collaboratively produce diverse and semantically consistent new samples. Experimental results show that CaDO significantly outperforms the strongest of five data cleaning baselines in 16 out of 18 experimental settings, improving downstream model accuracy by up to 9.1%. Moreover, at least 95% of the samples flagged by CaDO are defective, and on CLadder, CaDO incorrectly modifies 92.2% fewer correct labels than an LLM-based judge, demonstrating its effectiveness and reliability. Our code and data are publicly available at https://github.com/thisissyu/cado.
BibTeX
@misc{yu2026cado,
  author = {Songli Yu and Qi Luo},
  title = {{CaDO: Formal Verification-Guided Multi-Agent Framework for Causal Dataset Repair and Generation}},
  year = {2026},
  eprint = {2611.24406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24406},
}
Figure