Translating Interactive Workflows into Reproducible Code

56 concepts38 papers1 proposal

When working with data, users face a fundamental trade-off between two modes of interaction: the immediacy of graphical user interfaces (GUIs) and the reproducibility of programmatic scripting.

GUIs, spreadsheets, and visual profiling tools are highly intuitive. They offer immediate visual feedback and require low cognitive load, making them ideal for exploratory data analysis. However, manual interactions are ephemeral; they cannot be easily documented, shared, scaled to larger datasets, or re-executed when the underlying data changes. Conversely, writing code in languages like Python or SQL provides deep transparency and reproducibility, but it introduces a steep learning curve and frequently interrupts the user's creative or analytical flow.

To bridge this gap, a powerful paradigm has emerged: automatically translating interactive user actions, demonstrations, and examples into executable, reusable code. By capturing user intent through direct manipulation and passing it to underlying synthesis engines, modern systems allow users to build robust computational pipelines without manual coding.


The Spectrum of Interaction-to-Code Translation

Systems that convert user interactions into code generally operate along a spectrum, ranging from direct API logging to complex program synthesis.

[Direct API Logging] ----------------> [Programming by Demonstration] ----------------> [Programming by Example]
Logs explicit GUI actions              Generalizes observed user behaviors             Infers logic from input-output pairs
as sequential code scripts.            into reusable scripts or rules.                 using constraint solvers and DSLs.

1. Direct API Logging and Provenance Tracking

The most straightforward way to bridge the interactive-programmatic divide is to ensure that every graphical action is backed by a formal programmatic API. When a user clicks a button, adjusts a slider, or filters a column, the system intercepts this action and appends the corresponding code snippet to a background script.

2. Programming by Demonstration (PBD)

While logging records exact actions, Programming by Demonstration (PBD) aims to generalize a sequence of observed manual actions into a reusable macro or rule.

As Martin Hirzel explains in Low-Code Programming Models, PBD records a user's manual steps—such as scraping a website or moving items between folders—and automatically generalizes them into an executable script that can handle new, unseen inputs. For instance, Mukul Singh et al. develop this concept for email organization in EmFORE: Learning Email Folder Classification Rules by Demonstration, synthesizing human-readable classification rules in real-time simply by watching a user manually sort emails. Similarly, the authors of Infrastructure for Rapid Open Knowledge Network Development leverage demonstration-based data ingestion to let non-technical users define web-scraping rules by highlighting elements on a webpage, which the system instantly converts into data-extraction scripts.

3. Programming by Example (PBE)

The most mathematically sophisticated approach is Programming by Example (PBE). Here, the user does not show the steps to complete a task; instead, they provide input-output examples or specify visual mappings, and a synthesis engine infers the underlying program.

As Cristina David and Daniel Kroening outline in Program synthesis: challenges and opportunities, this shifts the programming paradigm from specifying how to implement a task to specifying what the task entails.


Why It Works: The Mechanics of Program Synthesis

How can a system reliably guess the correct code from a few clicks or a single input-output example? The magic relies on restricting the search space using Domain-Specific Languages (DSLs) and Syntax-Guided Synthesis.

User Input                     Domain-Specific Language (DSL)
[ "John Doe" -> "J. Doe" ]  +  [ Concat(), Substring(), Upper() ]
            \                     /
             \                   /
              v                 v
         +---------------------------+
         |  Synthesis Engine         |
         |  (Prunes invalid paths via|
         |   Counter-Example Guided  | ----> Generated Code:
         |   Inductive Synthesis)    |       Concat(Left(In, 1), ". ", Right(In, 3))
         +---------------------------+

The Role of Domain-Specific Languages (DSLs)

If a synthesis engine had to search through every possible Python or C++ program to satisfy a user's example, the problem would be computationally intractable due to an infinite search space. To make synthesis instantaneous, researchers restrict the engine to a highly specialized DSL containing only the operations relevant to the task.

In his seminal paper, Automating string processing in spreadsheets using input-output examples, Sumit Gulwani introduces this concept for spreadsheet text manipulation (the foundation of Microsoft Excel's Flash Fill). By restricting the language to basic string operations like Substring, Concat, and regular expression token matching, the system can rapidly search the space of valid programs and find one that satisfies the user's input-output pairs.

This approach generalizes across many data domains:

High Sample Efficiency

Because the DSL heavily constrains what programs are possible, these systems achieve remarkable sample efficiency, often requiring only a single example to find the correct logic. For instance, Jeevana Priya Inala et al. demonstrate in WebRelate: Integrating Web Data with Spreadsheets using Examples that a synthesis engine can accurately infer URL structures and web-scraping extraction logic from just one row of a spreadsheet, immediately automating the data join for thousands of remaining rows.


Synthesis as an Educational Scaffold

Beyond automation, translating interactions into code serves a vital pedagogical purpose: it acts as an educational scaffold. By immediately displaying the code generated by a visual action, systems help users learn formal programming syntax implicitly.

In Demo: Synthesis-Enabled Live Coding on the Web, Catherine Ji et al. present a live-coding music environment where clicking a traditional drum-machine GUI automatically generates text-based music code. This allows beginners to observe the real-time relationship between intuitive visual manipulation and formal programming logic, easing their transition to manual coding.

This real-time feedback loop is mirrored in scientific software documentation. Gareth A. Tribello et al. describe in PLUMED Tutorials: A collaborative, community-driven learning ecosystem how parsing and rendering scientific software input files directly within interactive tutorials provides immediate visual feedback on code validity, helping users master complex syntax by linking keywords directly to interactive explanations.

By turning ephemeral clicks into permanent, readable code, interactive-to-scripted workflows democratize programming. They eliminate the "syntax tax" of data science, allowing users to focus on what they want to achieve while the synthesis engine handles how to write it.

Key Foundational Work

Mayavi: Making 3D Data Visualization Reusable (2008) — contributed 1 concept elements to this direction.
Generating photo manipulation tutorials by demonstration (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_programming_by_example_6ef98378,
  title = {programming by example},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programming-by-example/rdr_6ef983788c8b4f21a6d6bd49a22abd45},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_programming_by_example_6ef98378_v1,
  title = {programming by example},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/programming-by-example/rdr_6ef983788c8b4f21a6d6bd49a22abd45/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake