AI code agents that analyze data in large, messy file systems mostly fail silently. They open the wrong file, join on a column that only looks right, or average a categorical code. No crash flags it, and the final number is simply wrong. A natural fix is to train a step verifier: a model that scores each step of an agent's trajectory and steers the agent away from bad ones. Such a verifier needs step-level labels, which nobody has at scale, so labels are produced by other models. The current approach has LLM judges vote and keeps the steps they agree on. That approach has a known blind spot. When the judges share a misunderstanding, they agree confidently and wrongly.
This proposal adds a second, differently-wrong source of labels: cheap symbolic checks. These are automatically built semantic profiles of files and columns, plus precondition rules over the agent's workflow, such as "this column is an identifier, so it cannot be averaged" or "this file's entities do not match the task's entities." The key design choice is to treat these checks as one-sided. When a check fires, it is strong evidence that the step is bad. When it stays silent, it says almost nothing, because the rules can never cover an open world. We claim that consensus labels plus one-sided symbolic negatives give a cleaner training set than either source alone, because the two sources make errors that are largely uncorrelated. We further claim that this advantage grows as the pool of pseudo-labeled trajectories grows, which is exactly where unfiltered training degrades.
Why this intersection, and what already exists
The scale problem is concrete. CoDA-Bench builds Linux sandboxes averaging 980 files and roughly 43 GB per environment, drawn from Kaggle communities. The best reported agent, Mini-SWE-agent with GPT-5.5, reaches 61.1% execution accuracy overall and 49.6% on the hard subset, CoDA-HARD. The best agents pick the wrong data file in about 20% of tasks, and supplying the correct files raises accuracy by 25–28%. This motivates the proposal. The dominant failure is semantic (which data means what), not syntactic, and it gets worse as environments grow.
Semantic workflow validation supplies the method we reuse. Simple-ML maps raw columns to domain concepts so that meaningless operations are blocked. ExeKGLib encodes machine-learning pipelines as knowledge graphs and checks preconditions with a shape-constraint language before execution, on a 2.74-million-record industrial dataset. Both assume a curated schema and use the checks as hard gates. We instead generate profiles automatically for uncurated directories and use check outcomes as noisy training signals rather than gates. Bel Esprit runs an LLM "inspector" for syntax and semantics, and it concedes that the inspector cannot verify custom code. IntelliGraphs shows that standard embedding models reach 0% logically valid novel graphs on real-world data. That result constrains the design: semantic validity has to come from explicit rules, not from embedding geometry.
The closest existing work on pseudo-label noise is DC-W2S. It trains step-level reward models from LLM pseudo-labels, keeping steps that satisfy both agreement across weak judges and consistency with neighbors in embedding space. It reports 68.5% average F1 against 60.4% for plain fine-tuning on PerturbQA, and a "less-is-more" effect in which a curated 100k subset beats the full 351k. DC-W2S states that it fails when all supervisors share a conceptual error. VeriSimpl documents the same shared-misinterpretation problem in LLM self-verification and fixes it with solver-based checks, raising average accuracy from 56.6% to 65.5% with GPT-4o.
Our proposal differs from DC-W2S in three ways. Its second filter is symbolic rather than geometric. That filter is asymmetric by construction. And it is applied to data-agent workflows rather than biological reasoning. It differs from ExeKGLib and Simple-ML because the rules feed a learned verifier that must generalize beyond what the rules encode.
Core Claim
Let $c(s) \in [0,1]$ be the consensus score of step $s$ from multiple LLM judges. Let $V(s) \in \{\text{reject}, \text{silent}\}$ be the output of the symbolic checks. Let $\pi$ be the precision of a reject, estimated on a small audited set. The training label is
$$\tilde{y}(s) = \begin{cases} 0 \text{ with weight } \pi & \text{if } V(s) = \text{reject} \\ c(s) & \text{if } V(s) = \text{silent} \end{cases}$$
We keep only steps where the label is confident under this rule. Silence never upgrades a step. The testable claim has three parts. First, on audited steps, judge errors and validator rejects are only weakly correlated. Second, because of that, the combined labels have lower false-positive rates than consensus alone, specifically on "plausible but semantically wrong" steps such as wrong-file selection. Third, a verifier trained on these labels improves agent execution accuracy on held-out communities. The gain over consensus-only training should widen as the trajectory pool grows from tens of thousands to hundreds of thousands of steps.
Research Program
The first component is a semantic profiler that works within the input/output limits CoDA-Bench documents (agent performance falls sharply above about 3 GB of data). For every file, it reads headers, schema, and a bounded row sample. It assigns columns coarse semantic types: identifier, categorical code, quantity with unit, timestamp, free text. It also extracts entity signatures, such as which keys and value ranges appear. The profiles are stored in a metadata vocabulary in the spirit of Simple-ML's dataset profiles. Building them costs a fixed sample per file regardless of file size.
The second component is a small rule library over agent actions. It covers type-operation compatibility (no arithmetic on identifiers), join-key compatibility (matching semantic types and overlapping value ranges), task-file relevance (entities named in the task must appear in a file's signature), and pipeline preconditions in the style of ExeKGLib (fit before predict, no target leakage across splits). Each rule runs on parsed agent code and observed outputs, not by re-executing arbitrary scripts.
The third component is the labeling and training loop. Agents generate many trajectories on training communities. Several LLM judges score each step. The validators run on each step. Labels are combined as above, with $\pi$ estimated per rule family from a few hundred human-audited steps. A step verifier is then fine-tuned on the result. Checkpoint selection uses no target labels: we follow Pseudo-Label Validation, which shows that an under-regularized auxiliary "imputer" model can score candidate models on unlabeled data, and we compare this against selection on source communities. At test time the verifier reranks candidate steps and trajectories (best-of-N) for a fixed agent.
The fourth component is a scale study. We vary the number of pseudo-labeled steps and environment size, measured in files per sandbox, and track when uncurated training stops helping. We also track whether symbolic curation moves that point.
Baselines
The naive reference is the agent without any verifier, which is Mini-SWE-agent with a fixed backbone run on our held-out communities. For calibration, the cited number is 49.6% execution accuracy on CoDA-HARD and 61.1% overall with GPT-5.5 (CoDA-Bench). Because we may not run that exact backbone, we will re-run the agent with our chosen model on our split and treat that measured number as the bar.
The second reference is a verifier trained on all pseudo-labeled steps with raw judge scores and no filtering. This tests the "scale absorbs noise" position directly.
The strongest competitor is DC-W2S-style dual consensus: judge agreement plus embedding-neighborhood consistency, reimplemented with code and profile embeddings in place of protein and cell features. It is the only published method that curates step-level pseudo-labels at scale and shows less-is-more. Its reported result is 68.5% average F1 on PerturbQA. No reported result was found on Lacuna for DC-W2S on CoDA-Bench or any data-agent benchmark. All verifiers share the same trajectory pool, judge models, verifier architecture, training steps, and hyperparameter search budget. Beating it means two things on held-out communities, both with confidence intervals from multiple seeds: higher step-level F1 on audited steps, and a statistically significant gain in execution accuracy under identical best-of-N budgets.
Two baselines represent the validators used directly rather than learned from. In the first, the symbolic checks act as a hard runtime guardrail with no learned verifier, as in ExeKGLib's pre-execution gating. In the second, an LLM inspector re-checks the agent's steps, as in Bel Esprit. Two ablations isolate the key ingredient. One treats validator silence as a positive label (symmetric use), which tests whether one-sidedness matters. The other removes judges and trains on validator labels alone, which tests whether consensus contributes anything.
Why not just…?
Why not just use the symbolic checks as a runtime guardrail? Rules catch only what they encode, and the open-world limitation of fixed schemas is exactly what OrchestrationBench and Bel Esprit acknowledge. A learned verifier can generalize from rule-caught examples to similar errors the rules miss. The guardrail baseline settles this. If it matches the trained verifier on held-out communities, the training step adds nothing, and we will report that.
Why not just use stronger LLM judges or self-verification? VeriSimpl shows that LLM verifiers repeat the generator's misreadings, and DC-W2S states that consensus fails under shared errors. Our decisive measurement is the error correlation between judges and validators on audited steps. If stronger judges already drive false positives on wrong-file steps toward zero, the symbolic source is redundant.
Why not just put the semantic file profiles in the agent's prompt? This is the most serious threat to the data-discovery part of the story. CoDA-Bench's oracle-file gain suggests that better file information alone could recover much of the gap. We include a profiles-in-context agent as a baseline, and we also run our verifier on top of it. If the verifier adds nothing on top of profile-aware prompting, we narrow the claim to non-discovery errors such as joins, types, and leakage, and say so.
Why not just weight pseudo-labels by conformal prediction sets? Adaptive Conformal Guidance reports gains of up to 5.98% over FixMatch-style thresholding. However, its coverage guarantee weakens under distribution shift, as COLA notes, and held-out Kaggle communities are shifted by design. We add conformal weighting of judge scores as a variant of the consensus baseline, so that the comparison isolates the symbolic source rather than the weighting scheme.
Proposed Experiments
The main testbed is RUC-DataLab/CoDA-Bench, split by community so that verifier training and evaluation never share environments. We report execution accuracy on full and CoDA-HARD held-out tasks, the file-identification failure rate, and step-level F1 and false-positive rate on roughly 1,000 human-audited steps.
The first decisive check is error correlation. If validator rejects and judge false positives are strongly correlated on audited steps, the core premise fails and we stop. The second is the one-sidedness ablation. The claim requires that symmetric use of the validators be worse than one-sided use. The third is the scale curve. We train every labeling scheme at several pool sizes and test whether the gap over dual consensus widens with scale, as the less-is-more result predicts if noise grows with pool size. The fourth is model selection. We check whether pseudo-label-based checkpoint selection beats selection on source communities, echoing the 73% versus 53% Camelyon17 result in Pseudo-Label Validation.
As a secondary transfer check, we swap the rule library for solver diagnostics and train a step verifier on optimization-modeling trajectories from NL4Opt and IndOR. That test shows whether one-sided symbolic labels help beyond data agents. The reference point there is VeriSimpl's 65.5% average accuracy.
References
- CoDA-Bench: Can Code Agents Handle Data-Intensive Tasks?
- Simple-ML: Towards a Framework for Semantic Data Analytics Workflows
- ExeKGLib: A Platform for Machine Learning Analytics Based on Knowledge Graphs
- Bel Esprit: Multi-Agent Framework for Building AI Model Pipelines
- IntelliGraphs: Datasets for Benchmarking Knowledge Graph Generation
- DC-W2S: Dual-Consensus Weak-to-Strong Training for Reliable Process Reward Modeling in Biological Reasoning
- VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification
- Pseudo-Label Validation for Unsupervised Domain Adaptation
- OrchestrationBench: LLM-Driven Agentic Planning and Tool Use in Multi-Domain Scenarios
- Adaptive Conformal Guidance for Learning under Uncertainty
- Conformalized Hierarchical Calibration for Uncertainty-Aware Adaptive Hashing