Teaching Procedure Parsers to Read Between the Lines

People who write instructions leave things out. "Add the milk. Stir until smooth. Pour into the pan." The second and third sentences never say what gets stirred or poured. The answer is a mixture of flour and milk that has no name anywhere in the text. A program-generated training corpus does the opposite: it states every participant, because the generator knows the full world state and has no reason to hide it. This note argues that the gap between synthetic and real procedural text is mostly this explicitness gap, not a vocabulary gap. A model trained on fully explicit synthetic text learns a shortcut: the answer to "what is being acted on?" is always a word in the current sentence. Real text breaks that shortcut, most visibly on entities that are never mentioned or that only come into existence through mixing and transformation.

We propose two pieces that address this together. The first is a parser that turns each instruction step into a small executable action frame (an action, who does it, what it acts on, what it produces). Its decoder chooses among three sources for each output token: it can write a word from its vocabulary, copy a word from the sentence, or point at an entry in a running memory of entities, including unnamed intermediates the parser itself created in earlier steps. The second is a synthetic-data recipe called simulator-licensed ellipsis. A procedural simulator produces text paired with state, and we then delete argument mentions exactly where the simulator's own state makes the missing participant uniquely recoverable. Deleting more than that would leave examples no model could answer; deleting less leaves the shortcut in place. The claim is that this pairing closes much of the synthetic-to-real gap on real procedural benchmarks. It should do so specifically on the unmentioned-entity decisions that prior work identifies as the dominant failure.

Background and where the gap actually is

Entity state tracking means following objects through a text: whether each object exists, where it is, and what it has turned into. Prior work has already located the hard cases. On ProPara, a benchmark of human-written science-process paragraphs, Dalvi et al., 2018 showed that models near-perfect on the synthetic bAbI tasks collapse on real text: EntNet reaches 26.07% macro accuracy against 45.08% for their ProGlobal model and roughly 80% for humans. They also found that 37% of the remaining errors are missed implicit creations and destructions. Rajaby Faghihi et al., 2023 feed TRIPS semantic parses, a symbolic parser's role labels such as Agent, Affected and Result, into neural trackers. This raises NCET from 66.0 to 68.8 document-level F1, but errors persist on "global entity decisions" about participants absent from the current sentence. Both papers motivate this proposal. The first shows that synthetic training transfers badly. The second shows that semantic-role structure helps but cannot recover missing arguments.

Two lines of work model the unnamed intermediates directly. Rim et al., 2023 annotate recipes in CUTL with "phantom nodes", unmentioned results of mixing or separating. They verbalize those nodes with GPT-3 paraphrases before tracking, and still reach only 58.01 Identity F1 on hidden entities. Khosla and Rosé, 2023 build ChemProp for chemical patents, where phrases like "the residue" bridge back to the reaction steps that produced them. End-to-end extraction falls from 0.86 F1 with gold entities to 0.69. Both works treat intermediates as something to annotate or paraphrase into text. We instead make intermediates first-class memory entries that the decoder can point to, so they never need a surface form.

The pointer-generator supplies the decoding method. In See et al., 2017, a learned gate mixes vocabulary generation with copying from the source. Bi et al., 2020 replace the gate with a discrete latent selector over several sources, trained with Gumbel-Softmax, a differentiable approximation to sampling a discrete choice. Li et al., 2021 copy jointly from questions and knowledge-base triples. All three assume the target token already exists somewhere: in the source text or in a static store. Our memory is neither. It grows as the parse proceeds, and its entries are created by the decoder's own earlier outputs.

There is also a real tension here. Wu et al., 2023 show on OpenPI-C that state descriptions such as "preheated" or "dissolved" must be generated freely. Joshi et al., 2020 penalize generation so that terminology gets copied. We resolve this by role rather than by global gate setting. Participants come from memory or the text. Predicates and attribute values come from the vocabulary.

The sim-to-real literature supplies the training lens. Li et al., 2023 improve synthetic-to-real LiDAR segmentation by learning to drop points from clean synthetic scans so they resemble real sensor dropout. Our analogue is that real writers "drop out" arguments. However, the dropout in text is not sensor noise. It follows a rule: people omit what is salient and unambiguous. That is why we license deletions with the simulator's state rather than learning them adversarially. Chen et al., 2020 counter forgetting during synthetic fine-tuning with a KL penalty against a frozen pretrained model; we treat that as a competing explanation to test. Awasthi et al., 2022 show that retrieve-and-edit synthesis makes text-to-SQL data far more diverse. That work targets lexical diversity in static queries, not omitted arguments in temporal procedures.

Core Claim

A parser that emits per-step action frames, whose participant slots are filled by pointing into a growing entity memory, will transfer from synthetic to real procedural text much better than parsers that copy only from text or generate entity names as strings. This advantage should appear only when the synthetic training text contains state-licensed ellipsis. Most of the gain should fall on decisions about participants that have no mention in the current step. Formally, at decoding step $t$ a selector gives probabilities $s_t^{voc}, s_t^{txt}, s_t^{mem}$ over the three sources, and

$$P(y_t) = s_t^{voc} P_{voc}(y_t) + s_t^{txt} \sum_{i: x_i = y_t} a_{t,i} + s_t^{mem} \sum_{j: m_j = y_t} b_{t,j},$$

where $a_{t,i}$ is attention over sentence tokens $x_i$, and $b_{t,j}$ is attention over memory entries $m_j$. The memory contains every entity introduced so far plus a reserved "new intermediate" entry. Pointing at that reserved entry allocates a fresh slot, initialized from the embeddings of the frame's inputs.

Research Program

The first artifact is a procedural simulator covering three domains that match the real benchmarks: cooking, bench chemistry (mixing, filtering, washing, drying, phase splitting), and science processes (creation, destruction, movement). The simulator samples a world, then a sequence of actions with preconditions and effects. It tracks every entity's existence, location and attributes, and creates a new entity whenever a mix or split occurs. It renders each step from varied templates. This mirrors how REFILL-style synthesis is grounded in executable structure, but here it applies to sequences.

Each step's gold output is an action frame whose Agent, Affected and Result roles follow the role inventory of Rajaby Faghihi et al., 2023. Role fillers are memory identifiers, not strings. Executing the frames reproduces the simulator's state grid, which in turn maps onto the ProPara grid, CUTL's identity and transformation links, and ChemProp's bridging relations.

The second artifact is simulator-licensed ellipsis. For each rendered argument mention, the simulator checks whether the participant is uniquely recoverable from the state. Examples of recoverable cases are: it is the only active intermediate; it is the Result of the previous step; or it is the only entity in the referenced container. Only recoverable mentions become candidates for deletion, or for replacement with a generic noun such as "the mixture" or "the residue". A single rate parameter sets the deletion probability. We set it to match an explicitness statistic measured on real data: the fraction of gold role arguments that have a surface mention in their own sentence. We can compute this statistic on the annotated portions of ProPara, CUTL and ChemProp. Matching the rate to it turns "make synthetic text look real" into a measurable quantity.

The third artifact is the three-source parser. It sits on a pretrained BART encoder-decoder with the selector of Bi et al., 2020. It is trained on synthetic data alone (zero-shot transfer) and on synthetic data plus 10%, 25% and 100% of each real training set. For ChemProp we additionally use its silver corpus.

The analysis question is why transfer fails, not only by how much. We will slice every error by whether the gold participant is mentioned in the step, mentioned earlier, or never mentioned (a phantom). This uses the decision-level error categories of Rajaby Faghihi et al., 2023.

Baselines

The naive reference point is a BART model fine-tuned on the real training data only. It linearizes the same action frames, writes participants as strings, and resolves them to entities by string match. It sets the floor and shows what synthetic data adds at all.

The strongest existing method on ProPara is the hybrid tracker of Rajaby Faghihi et al., 2023. NCET with TRIPS features reports 68.8 document-level F1. They describe CGLI with TRIPS features as the best on ProPara, but no numeric value for it was found in the file. We will rerun their released configuration on the same splits and real-data fractions. Beating it means exceeding 68.8 document F1 with full real data. It also means a significant gain on the global-entity slice, the category they report as unsolved. A gain confined to mentioned entities would not support our claim.

On CUTL, the reference is the hidden-entity result of Rim et al., 2023: 58.01 Identity F1 and 76.72 Aggregation F1. Their system relies on GPT-3 paraphrases of phantom nodes. We compare without any such paraphrasing. On ChemProp, the reference is the 0.69 end-to-end F1 of Khosla and Rosé, 2023, and the target is to narrow the gap toward the 0.86 gold-entity pipeline. On OpenPI-C, a secondary check of whether the vocabulary branch still produces open-ended states, the reference is the 8.26% cluster exact-match F1 of the BART model with entity memory and entity-conditioned generation from Wu et al., 2023.

Three ablations isolate the ingredients, all using the same backbone, synthetic corpus size, real-data fraction and hyperparameter search budget. The first removes the memory branch, leaving a two-source text-copy pointer-generator. The second keeps the memory but trains on fully explicit synthetic text. The third replaces licensed deletion with random deletion at the same overall rate. The proposal's central prediction is an interaction: memory pointing helps substantially more with licensed ellipsis than without it.

Why not just…?

Why not just fine-tune a large pretrained model end to end and let it name entities as strings? This is the default competitor, and it is included above as the naive baseline plus the no-memory ablation. Existing evidence suggests it struggles. A fine-tuned GPT-2 reaches only 5.72% cluster F1 on OpenPI-C, and explicit entity memory both raises that to 8.26% and cuts audited hallucinations by 47.5% (Wu et al., 2023). Strings also have no stable way to name an unnamed mixture, which is the phantom case. We found no reported result on Lacuna for prompted large language models on ProPara, CUTL or ChemProp. We will therefore run a few-shot prompted model emitting the same frames as an extra reference point and report it by error slice. If it matches our parser on the never-mentioned slice, the memory pointer is not needed and we will say so.

Why not just delete mentions at random? This is the most serious threat, because random dropout is far simpler and might teach the same lesson. Our argument is that random deletion creates unanswerable examples, which teach the model to guess the most recent entity. Licensed deletion only removes what the state determines. The random-deletion ablation, run at a matched rate, decides this. If it performs within noise of licensed deletion on the global-entity slice, the licensing machinery is unnecessary and the contribution reduces to "memory pointer plus ellipsis augmentation". We would report it that way.

Why not just paraphrase the synthetic text with a language model, or use retrieve-and-edit, to make it look real? Paraphrasing raises lexical diversity, which Awasthi et al., 2022 show helps text-to-SQL. However, we expect it to preserve or even increase explicitness, since paraphrasers tend to spell out referents. The dense paraphrases of Rim et al., 2023 move in exactly that direction. We will paraphrase the synthetic corpus with a pretrained seq2seq model and measure its explicitness statistic alongside downstream accuracy. If paraphrasing alone closes the gap, our diagnosis is wrong.

Why not just regularize against forgetting, as in Chen et al., 2020? Their KL penalty toward a frozen pretrained model is cheap to port. Running it on top of the no-memory, explicit-text baseline tests the competing explanation that the gap is forgetting rather than explicitness. That explanation predicts uniform gains across error slices. Ours predicts gains concentrated where mentions are missing.

Proposed Experiments

The first decisive check is diagnostic and needs no new model. We compute the explicitness statistic on fully explicit synthetic data and on each real benchmark. We then sweep the licensed-deletion rate from zero to beyond the real value and plot zero-shot ProPara and CUTL accuracy for the three-source parser. If the explicitness gap drives transfer, accuracy should peak near the real rate and fall at higher rates as recoverability breaks down. The two-source text-copy model should show little benefit at any rate, because it has no way to output an absent participant.

The second check is the factorial comparison: memory branch on or off, crossed with explicit, random-deletion, or licensed-deletion synthetic text, at each real-data fraction, on ProPara document F1, CUTL hidden-entity Identity and Aggregation F1, and ChemProp end-to-end F1. The prediction is that licensed ellipsis with memory pointing beats every other cell, with gains concentrated on never-mentioned participants. A low-data claim also has to hold: synthetic training plus 10% real data with our method should match or exceed the full-real-data NCET-plus-TRIPS result of 68.8 on ProPara. If it does not, the synthetic route is not paying for itself.

The third check concerns the tension with open-vocabulary state tracking. We inspect the selector's choices on OpenPI-C to confirm that attribute values come mostly from the vocabulary branch and participants from memory. A memory branch that suppressed fluent state generation would show up as a drop below the 8.26% reference.