Systems that turn a natural-language question into a database query ("which patients had a potassium test within two days after admission last year?") are rarely trained once and left alone. They are adapted again as new hospitals, schemas and question styles arrive. Each update risks quietly breaking what the parser already knew. The parts most likely to break are its temporal operators: "before", "first", "within N days", interval overlap. These are rare, structurally distinctive, and easy to confuse with one another. Rehearsal-based continual learning is the standard remedy for this kind of forgetting: keep a small memory of old training examples and mix them into each new update. But it was designed for image classes. It treats every stored example as an interchangeable point. The "boundary" variants, which nudge stored examples toward the decision boundary, do so with gradient steps in a continuous feature space, and that produces garbage when the example is a SQL string.
We propose to make replay respect the structure of temporal queries. Every stored query is reduced to an operator skeleton: its pattern of temporal operators with constants, tables and entities abstracted away, borrowing the normal-form idea from temporal query rewriting. The memory is balanced over these skeletons rather than over raw examples. "Boundary" examples are produced as small edits that keep the program well-formed: swap before for after, shift an interval endpoint, change first to last. Each edit is executed against the database, and we keep only the edits whose answer actually changes. The same memory also holds answerable and unanswerable questions per skeleton, so the parser's refusal detector can be recalibrated rather than silently drifting. The claim is that forgetting in continually adapted temporal parsers concentrates in specific operator skeletons. Replay that targets those skeletons with execution-verified contrast pairs should retain them better than random replay at the same memory budget, without hurting the parser's ability to refuse unanswerable questions.
Core Claim and What Is New
The motivating evidence comes from two temporal query interfaces that are trained, or retrained, without any protection against forgetting. EHRSQL is a clinical text-to-SQL benchmark over MIMIC-III and eICU in which 93.2% of questions involve time, through absolute, relative and interval templates. It pairs the parser with an entropy-threshold detector that refuses unanswerable questions. Both the parser and its threshold are fit once. Prog-TQA answers questions over temporal knowledge graphs by generating programs with operators such as FilterBefore, FilterFirst and GetYear. It repeatedly fine-tunes its generator on programs that happened to produce correct answers. It reports that some of these programs are "spurious", meaning they reach the right answer by a wrong path, and it has no mechanism to protect older operator patterns during those rounds. Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL is the closest existing combination of query parsing and continual learning. It adapts parsers to new domain knowledge from feedback but does not treat temporal operators as the unit that is forgotten, and it has no replay that respects the query grammar. Our proposal differs from it on exactly those two points, and it also evaluates refusal behaviour under shift.
The method combines ideas from two papers that disagree. Continual Learning with Tiny Episodic Memories shows that replaying one to three examples per class, kept in a per-class first-in-first-out buffer, beats heavier regularization methods. It reports a 7–17% average-accuracy gain over EWC and A-GEM. This constrains the design: if we cannot beat a well-balanced simple buffer, the extra machinery is unjustified. Retrospective Adversarial Replay argues the opposite. It reports that synthesizing replay points near the forgetting boundary cuts forgetting by nearly 40% and reaches 28.8% average accuracy on Split-miniImageNet. Its perturbations live in a continuous latent space, which is exactly what cannot be done for discrete programs.
Our reading is that the two results disagree about what a "class" is and where the boundary lies. For temporal queries, operator skeletons give a natural class for balanced buffering in the Tiny-Memories sense. Grammar-preserving edits that change the executed answer give a discrete, verifiable version of the boundary points used by Retrospective Adversarial Replay. The execution check is the novel ingredient, and it does two jobs. It guarantees that the contrast pair really differs in meaning. It also pushes back against spurious programs, because the model is trained to tell apart programs that differ in a single operator.
Skeletons come from the rewriting literature. Temporal Conjunctive Query Answering via Rewriting normalizes temporal conjunctive queries into a canonical form so that different surface forms of the same logical query coincide. Computing Compact Answers to Temporal Path Queries Using SQL maps temporal operators to a small set of SQL building blocks (range joins, window functions and interval coalescing, i.e. merging overlapping time intervals). We reuse this idea only as a way to canonicalize queries. A skeleton is the multiset of temporal building blocks, plus their nesting, after abstracting constants. We make no claim about rewriting efficiency.
Research Program
We first build skeleton extractors. For EHRSQL SQL, the extractor parses each query and maps its time predicates, ordering-plus-limit patterns, date arithmetic and nested time subqueries onto a fixed vocabulary of temporal building blocks. For Prog-TQA programs, the operator sequence is already explicit, so the extractor only abstracts entity and time arguments.
We then run a diagnostic study before proposing any method. We train a parser sequentially over a stream of tasks and measure forgetting separately for each skeleton. If forgetting is roughly uniform across skeletons, the core premise fails and the project reduces to evaluating ordinary replay.
The method itself has three parts. The memory holds a fixed budget of examples, allocated evenly across the skeletons seen so far. Within each skeleton it keeps a first-in-first-out queue. At each replay step, every stored example spawns up to $m$ edits drawn from a small library of operator flips and endpoint shifts. Each edit is executed, and we keep only those whose result set differs from the original. The edited program is paired with a templated question generated by EHRSQL-style templates or by a frozen language model paraphraser, and the pair is used as an extra training example. A second slice of the memory stores answerable and unanswerable questions for each skeleton. After each task, the refusal threshold is refit on that slice, following the observation in MAGE that detector failure under shift is often a threshold problem fixable with very few target examples, sometimes as few as four.
Proposed Experiments
We use two streams. The first is built from EHRSQL. The parser is trained on MIMIC-III questions ordered by temporal template family: absolute, then relative, then interval. A final task moves to eICU, which adds a schema shift on top of the template shift. The second stream is built from MultiTQ and CronQuestions. Questions are split into tasks by operator family, using the program structure from Prog-TQA, and fed to an iteratively self-trained program generator.
Metrics are reported per skeleton and then averaged over skeletons:
- Execution accuracy, and Hits@1 for knowledge-graph questions.
- Forgetting, defined as the best accuracy reached on a skeleton minus its final accuracy.
- Answerability F1 as defined in EHRSQL.
- False-refusal rate on old skeletons.
- Spurious-correct rate on EHRSQL, meaning the fraction of correct-answer outputs whose skeleton differs from the gold query's skeleton. This is measurable because EHRSQL provides gold SQL.
The training upper bound is retraining on all data seen so far. For reference points: EHRSQL reports best-baseline execution accuracy of "around 88–94%" and answerability F1 "over 90%" in its static, non-continual setting. Prog-TQA reports relative Hits@1 improvements of 60.9% on multi-constraint and 47.0% on single-constraint MultiTQ questions. No reported result found on Lacuna for any continual or sequential-task setting on EHRSQL, MultiTQ or CronQuestions. The comparison therefore has to be produced inside this study.
Baselines
The naive reference point is sequential fine-tuning with no memory, alongside the joint-retraining upper bound. These two bracket how much forgetting there is to recover.
The strongest existing competitor is balanced experience replay as in Continual Learning with Tiny Episodic Memories, run in two forms. One is a random reservoir buffer. The other is a first-in-first-out buffer balanced over EHRSQL's own template identifiers, which is a skeleton-like stratification that comes free with the dataset. This baseline is the real threat. Its reported advantage is 7–17% average accuracy over EWC and A-GEM on vision benchmarks, and the forgetting it leaves behind on Split-MNIST is about 13–17%, the figure that Retrospective Adversarial Replay cuts to about 7–11%.
A second competitor is the feedback-driven continual text-to-SQL approach of Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL. We re-implement it as closely as the paper allows, with simulated feedback derived from gold queries. No reported number on our streams was found on Lacuna.
Every method gets the same backbone, the same memory budget counted in stored examples, and the same hyperparameter search. Because edits create extra training examples, baselines are given a matched number of extra replay steps, so that we compare memory content rather than compute.
We isolate the key ingredient with ablations. The first is skeleton-balanced buffering with no edits. The second adds edits but drops the execution check, so edits that do not change the answer are still trained on. The third adds execution-checked edits but no detector slice. The method counts as better only if it reduces mean per-skeleton forgetting by a margin beyond seed variance (five seeds) relative to template-balanced replay at equal budget. In addition, its answerability F1 after the eICU shift must not fall below that of a threshold refit on the same number of target examples.
Why not just…?
Why not just retrieve similar past examples into the prompt of a large language model and skip fine-tuning? This is a serious threat. The evidence against it in the file is weak. The GAP parser's 4.7% zero-shot execution accuracy on EHRSQL comes from an older parser without retrieval. Prog-TQA does report that plain in-context prompting struggled with combined temporal constraints. We therefore include a retrieval-augmented few-shot baseline that draws exemplars from the same memory and uses the same language model. If it matches the replay-trained parser on old skeletons, the honest conclusion is that the memory matters more than the training method. Skeleton-balanced selection would then be the contribution, used for retrieval rather than replay.
Why not just use a simple balanced buffer, since Tiny Episodic Memories says that suffices? It may. That is why the skeleton-balanced, no-edit ablation and the template-balanced buffer are both included. The added complexity of edits is justified only if it lowers forgetting on rare, easily confused operator pairs. Before/after flips and first/last flips are where we expect the gain. We will report results for those pairs separately rather than averaged into the overall score.
Why not just recalibrate the refusal threshold with a handful of new-domain examples, as MAGE suggests, and leave the parser alone? Recalibration addresses refusal drift but not parser forgetting, so it is complementary rather than a substitute. We run it as the detector baseline. If the per-skeleton detector slice adds nothing over a global threshold refit, we drop it and keep MAGE-style recalibration.
Decisive Checks
Three outcomes settle the proposal. The first is whether forgetting concentrates in a minority of skeletons in the no-memory run. If it does not, the premise is wrong. The second is whether execution-checked edits beat unchecked edits. If they do not, the discrete "boundary" analogy adds nothing beyond data augmentation. The third is whether the spurious-correct rate on EHRSQL falls under our method relative to template-balanced replay. That would be direct evidence that contrastive operator edits teach operator meaning rather than answer matching, which is the failure Prog-TQA reported but left unaddressed.
References
- EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records
- Self-Improvement Programming for Temporal Knowledge Graph Question Answering
- Continual Learning of Domain Knowledge from Human Feedback in Text-to-SQL
- Continual Learning with Tiny Episodic Memories
- Retrospective Adversarial Replay for Continual Learning
- Temporal Conjunctive Query Answering via Rewriting
- Computing Compact Answers to Temporal Path Queries Using SQL
- MAGE: Machine-generated Text Detection in the Wild