Programmatic Input Synthesis for Automated Software Testing

21 concepts16 papers2 proposals

When evaluating or fuzzing software, the quality of your test inputs determines the depth of the bugs you can uncover. With the rise of Large Language Models (LLMs), a natural first instinct is to ask a model to generate these inputs directly—for instance, asking it to output a complex JSON object, a massive array for a stress test, or a raw binary file.

However, direct data generation hits a hard wall for two fundamental reasons: context window limitations and structural invalidity.

First, large-scale test inputs quickly exhaust a model's context window. As noted by Peng et al. in COFFE: A Code Efficiency Benchmark for Code Generation, generating massive datasets directly as raw text creates an unsustainable token footprint. He et al. echo this challenge in HARDTESTGEN: A High-Quality RL Verifier Generation Pipeline for LLM Algorithimic Coding, pointing out that complex data structures often exceed a model's native generation capabilities when synthesized as raw text.

Second, LLMs struggle to maintain strict structural invariants over long sequences. For example, generating a valid video file or a highly specific binary format requires precise byte alignments that text-prediction models frequently hallucinate.

To solve this, a powerful paradigm has emerged: Programmatic Input Synthesis. Instead of forcing a model to generate the raw data, we prompt it to generate the program that creates the data.

Code as a Data Abstraction

Why does shifting from direct data generation to code generation work so well? The core intuition lies in leveraging the model's strong programming priors to delegate constraint enforcement.

When an LLM writes an executable script rather than raw data, it offloads the complexity of maintaining data formats and logical constraints to the execution environment. As Kaibo Liu demonstrates in LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programs, this programmatic approach ensures significantly higher input validity because the interpreter or compiler handles the strict formatting rules, not the language model. The model writes the rules; the execution environment enforces them.

This abstraction is particularly effective for non-textual and binary formats. In Low-Cost and Comprehensive Non-textual Input Fuzzing with LLM-Synthesized Input Generators, Zhang et al. show that LLMs are far more adept at writing scripts that call domain-specific libraries (such as graphics or multimedia libraries) than they are at generating raw binary streams. The code acts as a high-level interface, transforming a difficult byte-generation problem into a structured logic problem.

Similarly, when dealing with complex system configurations or command-line options, Wang et al. show in ProphetFuzz: Fully Automated Prediction and Fuzzing of High-Risk Option Combinations with Only Documentation via Large Language Model that generating executable scripts to produce valid binary files bridges the gap between the text-based nature of LLMs and the rigid structural requirements of downstream software.

Infinite Scaling and Stress Testing

Another major advantage of programmatic synthesis is token compression. A model can write a three-line Python loop or a list comprehension that instantiates a million-element array. By using these compact programmatic representations, the authors of COFFE demonstrate that models can generate massive "stressful" test cases while maintaining a minimal token footprint during the generation phase.

Furthermore, code allows for the introduction of dynamic scaling and seeded randomization. Instead of hardcoding a single test case, the model can synthesize an entire generator class. As shown by Ye et al. in Themis: Automated Constraint-Aware Test Synthesis Framework for Code Reinforcement Learning, an executable generator class can use randomization to scale test suite creation infinitely, automatically producing a diverse spectrum of typical, edge, and adversarial cases while strictly adhering to semantic constraints.

┌──────────────────────────────┐
│      Large Language Model    │
└──────────────┬───────────────┘
               │
               ▼ Synthesizes
┌──────────────────────────────┐
│  Executable Generator Code   │ ◄── Tiny token footprint
└──────────────┬───────────────┘
               │
               ▼ Executes with Randomization
┌──────────────────────────────┐
│  Infinite, Valid Test Cases  │ ◄── Massive data scale
└──────────────────────────────┘

This capability is vital for evaluating algorithmic efficiency and edge-case correctness in competitive programming. In HardTests: Synthesizing High-Quality Test Cases for LLM Coding, He et al. employ language models to write executable scripts that generate large-scale inputs, ensuring they maintain structural validity and strictly adhere to problem constraints.

Similarly, Wang et al. in CodeContests+: High-Quality Test Case Generation for Competitive Programming utilize a Generator agent to write C++ programs that explicitly cover varied data sizes and special data types, ensuring comprehensive test coverage that would be impossible to achieve via static text generation.

This philosophy extends beyond testing to data manipulation: Nobari et al. demonstrate in TabulaX: Leveraging Large Language Models for Multi-Class Table Transformations that synthesizing reusable Python functions from a few examples is far more scalable and transparent across large datasets than performing row-by-row data manipulation directly within the model.

Closing the Loop with Automated Verification

Generating diverse inputs is only half of the equation; an automated testing framework must also know if the software behaved correctly. Programmatic input synthesis naturally extends to generating the validation logic alongside the data generator.

Rather than outputting static input-output pairs, modern frameworks prompt models to generate comprehensive test harnesses. As detailed by Liu et al. in HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning, an executable test script combining an input generator with a validation function allows for the discovery of deep logical errors and corner cases that static test sets omit.

This dual-component architecture is also central to rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset, where Liu et al. implement a mutual verification pipeline that generates both a data generator and a constraint validator for each problem, ensuring rigorous testing across multiple scales.

Once these programmatic test suites are generated, they enable powerful automated verification strategies like differential testing and execution agreement. For example, Chen et al. introduce CodeT: Code Generation with Generated Tests, a framework that prompts LLMs to generate synthetic unit tests and then uses a "dual execution agreement" across candidate solutions to determine functional correctness without human intervention.

This programmatic paradigm builds on a long lineage of automated testing concepts. Historically, tools like OCAT by Jaygarl et al. captured real-world object instances from active program executions to bypass the manual specification of complex invariants (OCAT: object capture-based automated testing), while Wang et al. derived executable acceptance tests from natural language specifications via constraint solving over use case paths (Automatic Generation of Acceptance Test Cases From Use Case Specifications: An NLP-Based Approach).

Even in environments where active execution feedback is entirely unavailable, generative frameworks can be trained to respect these boundaries. As shown by Dai et al. in Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration, energy-based modeling can learn to produce valid inputs for complex software systems purely from data patterns, proving that capturing structural validity is the ultimate key to automated quality assurance.

Key Foundational Work

Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration (2020) — contributed 1 concept elements to this direction.
OCAT: object capture-based automated testing (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_programmatic_test_generation_ffbcaa36,
  title = {programmatic test generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programmatic-test-generation/rdr_ffbcaa36d9e64f08a918a410dc545d33},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_programmatic_test_generation_ffbcaa36_v1,
  title = {programmatic test generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programmatic-test-generation/rdr_ffbcaa36d9e64f08a918a410dc545d33/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake