TL;DR
DataTrace is a new benchmark that shifts the evaluation of LLM-based data agents from "did they get the right answer?" to "did they get it for the right reasons?" By analyzing the evidence path agents take to find root causes in data pipelines, the researchers discovered that a significant portion of "correct" diagnoses are actually unsupported lucky guesses—and that training agents to show their work significantly boosts their actual diagnostic performance.
What Problem Does This Paper Address?
As we delegate complex data engineering tasks—like diagnosing why a dashboard is showing incorrect revenue—to LLM-based "data agents," we face a trust gap. Traditional benchmarks are outcome-oriented: they check if the agent's final answer matches the ground truth.
However, in a complex data pipeline, an agent might identify the correct faulty table simply because its name sounds relevant to the symptom, rather than by actually tracing the data lineage or inspecting the query plans. This is the "lucky guess" problem. If an agent provides a correct-looking diagnosis without establishing the evidence, it is untrustworthy in a production environment. The authors address the lack of a structured, automated way to verify the reasoning trajectory of these agents across multi-artifact workspaces (SQL, logs, schemas, and dbt models).
Methods
The authors introduce two core components:
- DataTrace Benchmark: A framework that represents database environments as typed provenance graphs. Nodes represent artifacts (tables, columns, query plans, logs, etc.), and edges represent dependencies.
- Controlled Fault Injection: Instead of relying on expensive and inconsistent human annotations, the researchers mechanically inject 21 types of faults (e.g., null injections, incorrect join keys, schema changes) into working pipelines. This allows them to automatically derive the "gold" root cause and the specific "evidence path" required to find it.
- CoSE (Generator-Solver Protocol): A protocol where a "generator" creates reproducible tasks and an "external verifier" scores the agent's trajectory.
- New Metrics:
- RC@1: Standard root-cause accuracy.
- ER (Evidence Recall): How much of the necessary evidence path the agent actually inspected.
- CUR (Conditional Unsupported Rate): The percentage of correct answers that failed to meet a minimum evidence threshold (the "lucky guess" metric).
Datasets
The benchmark, DataTrace-Eval-v1, consists of:
- 83 runnable environments spanning nine industry sectors (finance, retail, etc.).
- Data sourced from BIRD-CRITIC, Spider2, dbt repositories, and mined production incidents.
- A frozen leaderboard split of 128 episodes across 13 environments for reproducible ranking.
- A total of 21 different fault operators covering source data, code, schema, and configuration errors.
Key Insights and Ideas
- The "Grounding Gap" is Real: Even top-tier proprietary models have a mean "Conditional Unsupported Rate" (CUR) of 18%. This means nearly one in five correct answers is essentially a guess unsupported by the evidence the agent actually looked at.
- Modality Neglect: Agents exhibit systematic biases in what they choose to inspect. They frequently consult logs and lineage but almost entirely ignore query plans and incident reports, even when those artifacts contain the smoking gun for the failure.
- Grounding as a Training Signal: Perhaps the most important finding is that grounding is actionable. By using the grounding signal as a reward in Reinforcement Learning (specifically Group-Relative Policy Optimization or GRPO), agents don't just become better at finding evidence—they actually become more accurate at finding the root cause.
- Distance and Modality Matter: Accuracy declines sharply as the "lineage distance" (the number of hops from the symptom to the cause) increases. "Silent drift" (where data is wrong but no error is raised) is significantly harder for agents to diagnose than explicit failures.
Key Results
- Baseline Performance: Seven proprietary models achieved an average root-cause accuracy of 48%.
- Training Gains: After in-loop training using the grounding signal, the share of grounded correct diagnoses rose from 83.7% to 94.6%.
- Accuracy Boost: Interestingly, this training also pushed raw root-cause accuracy from 29.2% to 36.7% on a matched held-out batch, proving that better investigation leads to better outcomes.
- Localization Failure: Fine-grained analysis showed that agents often name the right type of fault but blame the wrong object (e.g., identifying a null injection but pointing to the wrong table).
Limitations and Future Work
- Single-Path Verification: The current grader recognizes one mechanically derived evidence path. The authors acknowledge that multiple valid investigation paths may exist and plan to support this in the future.
- Causal vs. Visitation: The metric measures if an agent visited the evidence, not necessarily if the final answer causally depended on it. The authors proposed "counterfactual masking" (hiding evidence to see if the answer changes) as a future direction to sharpen this.
- Synthetic vs. Organic: While the benchmark uses real-world schemas, the injected faults may not perfectly match the frequency of natural production incidents.
- Human Validation: While a human-in-the-loop pipeline was designed, the full-scale human study remains incomplete.
Summary updated