A dialogue system has to keep a running record of what the user wants: the hotel area, the price range, the number of guests. The cheapest way to maintain that record is to edit it turn by turn ("insert area = north", "delete price"), not rewrite it from scratch. Edits are fast, but they create a specific weakness. A missed or wrong edit stays in the record for the rest of the conversation, because the only way to fix it is a later edit that happens to correct it. This note asks what happens when such an edit-based tracker is trained on one new domain after another, as deployed systems are. We conjecture that forgetting looks different there than in the systems the continual-learning literature studies. A model that has partly forgotten an old domain does not just make one more mistake per turn. It loses the corrective edits, the deletes and overwrites that undo errors, and every lost correction poisons all later turns.
We propose to measure this interaction directly and to attack it with one mechanism that serves two purposes. Each domain has a public schema, meaning its list of slots and allowed values. From that schema we procedurally generate synthetic dialogues whose correct edit sequences are known by construction. Replaying these dialogues while training on later domains is a form of rehearsal that stores no user data. Generating them with renamed slots and fresh values gives a test set that cannot have leaked into any model's training corpus. The claim is that replay built this way, and biased toward dialogues where the user changes their mind, recovers most of the closed-loop forgetting that per-turn metrics hide.
Core Claim
In continual dialogue state tracking (DST, the task of keeping that structured record of user goals), the loss caused by forgetting splits into two parts. The first is per-turn forgetting: how often the model emits a wrong edit when it is given the correct previous state. The second is propagation amplification: how much worse things get when the model instead conditions on its own previous, possibly corrupted, state. We measure both by evaluating every checkpoint twice, once teacher-forced (gold previous state supplied) and once closed-loop (the model's own state carried forward). We summarize the gap as an amplification ratio
$$A_k = \frac{1 - \mathrm{JGA}^{\text{closed}}_k}{1 - \mathrm{JGA}^{\text{forced}}_k},$$
where $\mathrm{JGA}_k$ is joint goal accuracy (the fraction of turns where the entire state is exactly right) on old domain $k$ after all later domains have been learned. A ratio $A_k$ near 1 means errors stay local. Large values mean they cascade.
We make three predictions. First, $A_k$ grows as domain $k$ recedes further into the training sequence. Second, among edit types, the corrective ones (delete, overwrite) lose accuracy fastest on old domains. Third, schema-driven replay that is dense in corrections lowers $A_k$ more than replay of the same size that contains only insertions.
None of the existing work makes this measurement. The continual DST methods TaSL (which protects parameter groups judged important to past domains) and Reason-of-Select distillation (which distills a large teacher's value-selection rationales into T5-small) both regenerate the full state every turn. In that setup errors do not carry forward through an edit chain, so the interaction we study cannot appear.
The edit-based trackers go the other way. Diable tracks state through table insert and delete operations and is 2.4x faster than full regeneration. LightWM applies set and remove patches to a structured memory. Both explicitly lack recovery from missed edits, and neither is trained across a sequence of domains.
Research Program
The generator. The first component builds on the logic of DyVal, which generates reasoning problems from graphs so that the answer is known by construction. Our version works per domain from a Schema-Guided Dialogue (SGD) schema. A scripted user-goal trajectory samples a sequence of state edits, with a controllable rate of corrections such as "actually, make it Friday" or "never mind the parking." An LLM then renders each scripted turn into a natural utterance. The labels come from the script, never from the LLM.
Paraphrase errors are the main risk. To catch them, a separate tracker decodes each rendered turn, and any turn where it disagrees with the script is discarded. We report the discard rate.
We also take seriously the warning from A²RBench that generator LLMs drift toward simpler content than requested. For that reason we measure whether generated dialogues match the real SGD dialogues in turn count, slots per turn, and correction frequency, rather than assuming they do.
The tracker and training sequence. The second component is a Diable-style edit tracker trained over the standard 15-domain SGD sequence used by TaSL and RoS. Training uses T5-small for direct comparability and a 7B model with LoRA adapters for the contamination question. When the model learns domain $j$, we mix in synthetic dialogues for domains $1..j-1$, generated fresh from their schemas. Nothing from real past users is stored.
Evaluation. Every checkpoint is evaluated teacher-forced and closed-loop on three test sets: the real SGD test split, schema-generated dialogues, and remapped schema-generated dialogues. The remapped set renames slots and substitutes unseen values. It borrows the symbolic-remapping probe from A²RBench, which found a 17.7% drop for frontier models on its abstract-reasoning tasks. Comparing the three test sets tells us whether a method's gains come from tracking the state or from memorized slot-value associations.
Following Granular Change Accuracy and Relative Slot Accuracy, we report accuracy on the changes made at each turn, broken down by edit type, alongside JGA. Joint accuracy alone hides where the errors start.
Baselines
Naive reference. The edit tracker is fine-tuned sequentially with no protection. We add two upper references: a cumulative full-state T5-small with the same naive training, and joint multi-domain training.
Strongest existing method. TaSL reports 62.1% average JGA on 15-domain SGD with T5-small, and an 8.8-point improvement in backward transfer (how much training on later domains changes accuracy on earlier ones). RoS reports 59.0% average JGA on the same setup. It cuts the first-task drop after 15 domains from 28% to 15%, but at the cost of throughput falling from 857 to 86 samples per minute. Both numbers come from cumulative trackers evaluated on static SGD.
We will rerun TaSL with its released configuration on our data splits and domain order, in two forms: on its original cumulative tracker, and transplanted onto the edit tracker. This separates the question of which state representation is better from the question of which forgetting defense is better. All methods get the same backbone, the same number of training steps per domain, and the same hyperparameter search budget, tuned on a held-out set of real SGD validation dialogues.
What counts as beating TaSL. Generator replay must match or exceed TaSL's average closed-loop JGA on the edit tracker, show lower $A_k$ for the oldest five domains, and keep Diable-level inference cost.
Real-data replay. An experience replay buffer of real past dialogues, matched to the synthetic buffer in number of turns, is the privacy-violating reference we hope to approach.
Key ablations. The first ablation isolates correction density: synthetic replay of identical size whose scripts contain only insertions. The second asks whether replay matters at all beyond simply seeing more turns: synthetic dialogues drawn from the current domain's schema only.
Why not just…?
Why not just regenerate the full state every turn, as TaSL and RoS do? Full regeneration removes the edit chain, so propagation amplification cannot arise. That is a real alternative. The cost argument for edits rests on Diable's 2.4x speedup and on the throughput collapse RoS reports. It is not a free win. Our design runs TaSL on both trackers. If the cumulative version achieves better closed-loop accuracy on old domains at acceptable cost, we will report that edit-based tracking is the wrong choice under continual training. That would itself be a useful finding.
Why not just replay a small buffer of real dialogues? For task-oriented dialogue involving appointments, payments, or health, storing raw past conversations is often not permitted, which is the constraint TaSL and LIFTR are built around. Real replay is also likely to perform very well. We therefore treat it as an upper reference rather than a straw man. The question the experiment answers is how much of its benefit schema-only replay recovers.
Why not just re-anchor the state periodically? One could regenerate the full state every $m$ turns to flush accumulated errors. This is a strong, simple fix for propagation, and we include it as a baseline swept over several values of $m$. It does nothing about per-turn forgetting, though. The teacher-forced numbers will show whether re-anchoring and replay address different parts of the loss and combine well.
Why not just evaluate on the static SGD test set? The cleanup in MultiWOZ 2.1 lowered TRADE's joint accuracy from 48.6% to 45.6%. DyVal found models with top static scores near zero on freshly generated tasks. A²RBench saw a 17.7% drop under symbolic remapping. For T5-small, contamination may be minor, and we will say so if the static and remapped scores agree. For 7B models trained on web text, a ranking flip between the real and remapped test sets would show that static numbers mislead.
Decisive Experiments
Does forgetting compound? Plot $A_k$ against how many domains ago domain $k$ was learned. If the curve is flat for the naive edit tracker, the compounding hypothesis is false, and the proposal reduces to a privacy-free replay study.
Are corrective edits forgotten first? Measure per-edit-type accuracy on old domains. If deletes and overwrites do not decay faster than insertions, the correction-density ablation should show no effect. We will report that outcome as such.
Does schema replay work? Compare generator replay against TaSL, real replay, and re-anchoring on closed-loop average JGA, backward transfer, and $A_k$. We judge success on the real SGD test set, so that gains are not artifacts of evaluating on our own generator. We then repeat the comparison on remapped dialogues. A method whose advantage disappears under remapping is relying on memorized surface forms rather than tracking state.
References
- TaSL: Continual Dialog State Tracking via Task Skill Localization and Consolidation
- Continual Dialogue State Tracking via Reason-of-Select Distillation
- Diable: Efficient Dialogue State Tracking as Operations on Tables
- LightWM: Training-Free Hierarchical Working Memory for Small Language Model Agents
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking
- Mismatch between Multi-turn Dialogue and its Evaluation Metric in Dialogue State Tracking
- Approximating Function Space Distance for Continual Learning in Transformers
- MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines