TL;DR
Large Language Models (LLMs) can generate sophisticated causal graphs to explain their reasoning, but they often fail to follow the rules they set for themselves. This paper introduces a "graph-contract audit" that treats these explanations as binding behavioral commitments, revealing a persistent gap between the logic an LLM reports and the decisions it actually makes.
What Problem Does This Paper Address?
In high-stakes fields like lending or college admissions, we don't just need AI to be accurate; we need it to be accountable. Causal graphs are a powerful tool for this because they make explicit claims: "Variable X should not affect the outcome," or "Information Y is only available after the decision and must be ignored."
However, there is a major gap in "explanation faithfulness." An LLM might provide a perfectly plausible-sounding causal graph (a rationale) that has nothing to do with its internal decision-making process. Current evaluations often focus on whether LLMs can reason with graphs provided to them, but this paper asks the opposite: Do LLMs actually follow the causal graphs they generate for themselves?
Methods
The authors propose a graph-contract audit, which treats a self-reported causal graph as a falsifiable contract. The process follows four steps:
- Report: The LLM is asked to provide a task-level Directed Acyclic Graph (DAG) for a specific decision task (e.g., loan risk).
- Derive: An evaluator parses this graph to find implied commitments. For example, if the graph shows that "Gender" is d-separated from the "Decision," the model has committed to gender being irrelevant.
- Test: The evaluator creates "matched profile pairs"—two identical profiles that differ only in the variable being audited (e.g., one has the variable omitted or swapped).
- Audit: The LLM is queried for predictions on these pairs in a fresh conversation where it cannot see its previous graph.
The primary metric is cMPS (noise-corrected mean probability shift). This measures how much the model's score changes due to the audited variable, after subtracting the baseline "noise" of the model's own decoding stochasticity.
Datasets
The study evaluates models across two structured binary decision tasks:
- Admissions: A synthetic task with a known structural causal model, allowing for diagnostic checks on whether the model's reported graph is even "correct" relative to the ground truth.
- Loan Default: A task resembling real-world underwriting. It includes "post-outcome" variables (like later missed payments) that are logically unavailable at the time of the initial loan decision, testing if models can respect temporal constraints.
The audit was performed on a "balanced core grid" of 192 cells, featuring 12 model variants (including families like Qwen, Llama, and Gemma) across various temperatures.
Key Insights and Ideas
- Rationales as Falsifiable Contracts: Instead of judging an explanation by how "plausible" it sounds to a human, we should judge it by whether the model's subsequent behavior contradicts it.
- The Accuracy-Faithfulness Paradox: A model can be highly accurate (high AUROC) while being completely unfaithful to its own reported logic. Accuracy is not a proxy for trustworthiness.
- Reminders are not Repairs: Explicitly showing the graph back to the model (a "graph reminder") or replaying the context moves the scores, but it doesn't necessarily make the model more consistent with its original rationale. It just steers the model into a different scoring regime.
- Role-Specific Failures: Models fail differently depending on the "role" of the variable. Violations regarding demographic fields (fairness) or post-outcome leakage (temporal logic) carry different weights for oversight than simple irrelevant feature violations.
Key Results
- Widespread Violations: Across both tasks, the audit found a persistent mismatch. In the admissions task, 48 of 96 cells licensed at least one test, and many showed significant "excess shifts" that couldn't be explained by noise.
- Scale Doesn't Solve It: Larger or newer models (like Qwen2.5 14B) still exhibited behavioral shifts (cMPS of 0.018 and MaxShift of 0.732) despite having decent predictive accuracy.
- Coverage as a Guardrail: The authors emphasize "coverage"—the fraction of a graph that is actually testable. A model that produces a graph with no testable implications might appear "faithful" (zero shift) simply because it didn't commit to anything falsifiable.
Limitations and Future Work
- Fixed Schemas: The audit currently requires a fixed set of variables and explicit decision times. It doesn't yet handle open-ended, multi-turn dialogues where variables might change.
- Local Interpretation: The tests are behavioral consistency checks, not population-level causal effect estimates. They tell you if the model is contradicting itself, not necessarily how it would perform on a real-world population.
- Schema Design: The method relies on the task designer to correctly identify which variables are "pre-decision" and "post-outcome." In complex, real-world workflows, defining this schema is a non-trivial first step.
The authors suggest that future work should move toward turning these causal explanations into practical accountability tools that can be integrated into socially consequential AI workflows.
Summary updated