TL;DR
CaDO is a multi-agent framework that uses symbolic causal engines to automatically diagnose, repair, and augment causal reasoning datasets. By translating natural language into formal queries, it replaces unreliable LLM "judgments" with verifiable computation, significantly improving the accuracy and reliability of downstream models.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly being trained on causal reasoning tasks, such as predicting the effect of an intervention or answering counterfactuals. Because real-world causal data is scarce, researchers often rely on programmatically generated datasets like CLadder or Corr2Cause.
However, these "synthetic" datasets are far from perfect. They frequently contain incorrect labels, flawed rationales, and structural imbalances (like "shortcut cues" where a specific word might accidentally correlate with a "yes" answer). Existing methods to clean this data usually involve "LLM-as-a-judge," which is inherently circular: you are using the very skill you want the model to learn to decide if the training data is correct. This leads to a "garbage in, garbage out" cycle where the judge's own reasoning errors are baked into the repaired dataset.
Methods
The authors propose CaDO (Causal-Engine-Guided Multi-Agent Data Optimization), a framework that shifts the source of truth from LLM opinions to executable formal logic. The process follows three main stages:
- Formal Causal Reasoning (The Verification Core): Instead of asking an LLM if an answer is right, a Formalizer Agent translates the natural language story into a formal causal query (a graph, a set of probabilities, and a specific question). This query is then handed to a Causal Engine—a symbolic program that applies the rigorous rules of causal inference to calculate the mathematically certain answer.
- Multi-level Diagnosis:
- Sample-level: A Diagnoser compares the original dataset's label and rationale against the engine's output. To ensure the formalization was correct, Critic Agents perform back-translation and independent checks to provide a "confidence tier" for every proposed repair.
- Dataset-level: A Profiler scans the entire dataset to find "coverage gaps" (missing types of questions) and "shortcut cues" (spurious correlations that models might exploit).
- Automated Repair and Generation: Based on a rule-based plan, the system applies six operators to fix labels, replace rationales, filter unanswerable items, and debias the text. Crucially, Explorer and Expander Agents generate new, diverse examples to fill gaps, which are then "round-trip" verified by the causal engine before being added.
Datasets
The framework was evaluated on three major causal reasoning benchmarks:
- CLadder: A dataset covering all three levels of the "Ladder of Causation" (association, intervention, counterfactuals).
- CauGym: A benchmark focused on multi-variable causal graphs with numerical probability answers.
- Corr2Cause: A dataset testing the ability to infer causal structures from correlations and independences.
The authors created "corrupted" versions of these datasets by injecting seven types of defects (e.g., flipping labels, reversing causal links, adding shortcut sentences) to rigorously test CaDO's ability to recover the ground truth.
Key Insights and Ideas
- Verification over Judgment: The most powerful idea is that for formal domains like causality, correctness is computable. By grounding the repair process in a symbolic engine, CaDO avoids the hallucinations and biases of LLM judges.
- Label-Blind Formalization: The Formalizer Agent never sees the original label. This prevents it from simply agreeing with the dataset and forces it to extract the underlying causal logic purely from the text.
- Precision Matters: In data cleaning, "doing no harm" is vital. The authors found that LLM judges often "break" perfectly good data (changing correct labels to wrong ones). CaDO's multi-agent critique system ensures that repairs are only made when there is "firm ground."
- Dataset-Level Health: Repairing individual rows isn't enough. By identifying and removing shortcut cues, CaDO prevents models from learning "cheats" and forces them to learn actual causal reasoning.
Key Results
- Superior Accuracy: Models fine-tuned on CaDO-repaired data outperformed those trained on the strongest baselines in 16 out of 18 experimental settings, with accuracy gains of up to 9.1%.
- High Reliability: At least 95% of the samples flagged by CaDO were truly defective. On the CLadder dataset, CaDO modified 92.2% fewer correct labels than a standard LLM-based judge.
- Robustness to Corruption: As shown in the sensitivity analysis, CaDO's performance remains remarkably stable even as the percentage of defective rows in the training data increases from 10% to 50%.
Limitations and Future Work
While highly effective, CaDO is currently limited by the scope of its Causal Engine. It can only verify and repair questions that can be expressed in the formal languages the engine understands. If a causal question involves nuances not captured by the current symbolic representation, the system must abstain.
Future work aims to extend the framework to handle more complex, naturally occurring causal defects and to expand the library of formal causal queries the engine can process. The authors also suggest that this "formalization-verification" loop could be applied to other structured domains like law, medicine, or software engineering.
Summary updated