Verifying Synthetic Data Through Code Execution

193 concepts135 papers1 proposal

Synthetic data generation has become a cornerstone for scaling the capabilities of large language models (LLMs). However, generating training data using LLMs introduces a fundamental paradox: if a model is prone to hallucinations and logical errors, its synthetic outputs will inevitably inherit those same flaws. Relying on another LLM as a judge often compounds the issue, substituting one subjective, noisy heuristic for another.

To break this cycle, a powerful paradigm has emerged: execution-based verification. Instead of asking a language model if an answer looks correct, researchers are forcing models to express their logic as executable programs, then leveraging deterministic environments—such as interpreters, compilers, databases, or simulators—to prove their correctness. By shifting the burden of verification from probabilistic models to deterministic logic, this approach ensures an uncompromised training signal.

The Core Mechanism: Programmatic Grounding

The intuition behind execution-based verification is straightforward: text is ambiguous and difficult to validate automatically, but code is binary—it either works or it breaks.

In code generation pipelines, the traditional approach of filtering data based on surface-level patterns or LLM grading is being replaced by rigorous runtime execution. For instance, Qian Liu et al. demonstrate in OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models that validating synthetic instruction data against unit tests eliminates hallucinations and ensures a high-fidelity dataset for supervised fine-tuning.

[LLM Generator] ──> Generates Code & Tests ──> [Isolated Sandbox]
                                                      │
                                            Executes & Checks Coverage
                                                      │
[High-Fidelity Dataset] <── Passes Verification ──────┘

To maximize the reliability of this filter, the verification process can be tightened beyond simple execution. In KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding, Zhangchen Xu et al. implement strict verification protocols that require full branch coverage during execution. This ensures that every logical path within the generated code is actively exercised and validated, filtering out subtle edge-case bugs that surface-level checks would miss.

Lifting Abstract Reasoning into Executable Code

How do we apply this deterministic filter to domains that aren't inherently about software engineering, such as mathematics? The answer lies in "lifting" abstract problems into generative code.

When generating synthetic mathematical datasets, human-like step-by-step reasoning (Chain-of-Thought) is notoriously difficult to verify automatically. To solve this, Elias Stengel-Eskin et al. introduce Executable Functional Abstractions (EFAs) in Executable Functional Abstractions: Inferring Generative Programs for Advanced Math Problems. Instead of generating a static word problem, the model is tasked with writing a Python program that encapsulates the underlying mathematical logic. This program must be capable of both sampling random parameters for the problem and programmatically calculating the correct solution. If the code executes successfully, it serves as an absolute mathematical guarantee of the training instance's validity.

Similarly, single-author works have leveraged this concept to scale data generation safely. Yifan Zhang introduces Template-based Data Generation in Training and Evaluating Language Models with Template-based Data Generation, using frontier models to author abstract meta-templates that are then instantiated and verified via programmatic code execution.

This programmatic grounding is particularly vital during knowledge distillation, where a small student model learns from a larger teacher. Biqing Qi demonstrates in PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning that filtering a teacher's reasoning paths through a Python interpreter prevents the student from learning from "unfaithful" reasoning—instances where the teacher arrived at a correct final answer through flawed intermediate logic. For formal mathematics, Yinya Huang et al. take this to the logical extreme in MUSTARD: Mastering Uniform Synthesis of Theorem and Proof Data by routing synthetic theorems and proofs through the Lean proof assistant, using strict compiler type-checking as an absolute judge of mathematical truth.

Verifying Agents and Domain-Specific Languages

The paradigm of execution-based validation naturally extends to autonomous agents and domain-specific languages, where the cost of a hallucination can be catastrophic.

For Large Action Models (LAMs) trained to interact with environments via APIs, text-based evaluation is insufficient. In xLAM: A Family of Large Action Models to Empower AI Agent Systems, Jianguo Zhang et al. validate synthetic agent trajectories by executing tool calls against live or simulated environments. This process instantly exposes hallucinated tool names or incorrect argument types, ensuring that the agent only learns from successful, valid interaction sequences.

We see identical patterns across other highly structured domains:

Why It Works: The Power of an Unyielding Oracle

The consistent finding across this research direction is that data quality trumps data quantity. In reinforcement learning and fine-tuning loops, a noisy reward signal allows models to exploit loopholes, memorize surface patterns, or learn degenerate behaviors.

Execution-based verification works because it provides an unyielding, objective oracle. If a generated solution contains a subtle logical flaw, the interpreter catches it. If an agent hallucinates an API parameter, the environment rejects it. By filtering out the noise before the training phase begins, models receive a pristine learning signal. The empirical evidence shows that training on a smaller, strictly verified dataset yields far greater generalization and reasoning capabilities, frequently allowing compact, open-source models to rival or surpass proprietary giants.

Key Foundational Work

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021) — contributed 3 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_execution_based_data_validation_345bd90e,
  title = {execution-based data validation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/execution-based-data-validation/rdr_345bd90ee098405095403f09f622b015},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_execution_based_data_validation_345bd90e_v1,
  title = {execution-based data validation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/execution-based-data-validation/rdr_345bd90ee098405095403f09f622b015/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake