Topic Episodes as the Unit of Memory for Spoken Dialogue State Tracking

Spoken task-oriented assistants have to remember what the user asked for across a long, messy phone conversation. The simplest approach, pasting every past turn into the model's input, works poorly. Real speech is full of fillers, back-channels ("I see"), self-corrections and speech-recognition errors, so the few turns that matter get buried. The obvious fix is to delete the "noisy" turns. On spoken data this backfires, because the turns that look like noise are often the important ones: a phone number spoken in three fragments, or a license plate corrected mid-sentence. We propose instead to organize history into topic episodes. These are contiguous stretches of conversation about one recurring subject, found by clustering turn embeddings across a corpus. The model sees the current episode word for word and earlier episodes only as compact state summaries, unless the conversation returns to them.

The contribution is a change in what gets pruned. Today, whole spans of irrelevant topic are kept verbatim, while individual turns that merely look irrelevant get deleted. We would do the reverse. The evaluation setting is SpokenWOZ, where the "spoken gap" is well documented. The added test is whether the same unsupervised episode structure holds up under domain shift, which is where current end-to-end speech trackers are known to break.

Core Claim

Dialogue state tracking (DST) means maintaining the set of slot values the user has specified so far, such as hotel area or train departure time. For spoken DST, the right unit for managing history is a topic episode, not a turn and not a fixed recency window. We make two concrete claims. First, keeping episodes contiguous protects values that are delivered across several turns, which turn-level filters tend to cut apart. Second, fragmentary turns that a clustering method marks as outliers should be attached to their neighbours, not discarded. We expect gains to concentrate on SpokenWOZ's cross-turn slots and on long dialogues, and to shrink on short or clean-text dialogues. If the gains do not follow that pattern, the claim is wrong.

Where This Sits

The motivating evidence comes from three papers. SpokenWOZ builds a spoken task-oriented dialogue benchmark and shows that its dual-modal baseline reaches only 25.65% joint goal accuracy (JGA, the fraction of turns where every slot is exactly right), rising to 28.15% when cross-turn slots are removed. That is direct evidence that aggregating information across noisy turns is a bottleneck. Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs connects a WavLM speech encoder to an LLM through a small trained connector and feeds the full raw history. It reports 42.17% JGA with Gemma-2-9B, but describes MultiWOZ transfer only as "less stellar." We reuse this architecture unchanged and alter only what history it receives. Intent-driven In-context Learning for Few-shot DST shows that masking history and conditioning on the extracted intent helps in clean-text few-shot DST. This supports the idea that less history can be better, but that method depends on a text intent parser that disfluent speech would break.

The closest prior work is Topic-Oriented Spoken Dialogue Summarization with Saliency-Aware Topic Modeling. It splits a neural topic model into "informative" and "other" topics, using gold summaries as supervision, and steers attention away from the "other" topics in call-centre summarization. We differ in three ways. Our method is unsupervised: episodes come from geometric clustering, not summary labels. It targets state tracking, where a single dropped digit is fatal. And it does not suppress noise-like material; it reattaches it. We argue that suppression, which is sensible for summarization, is the wrong design for spoken DST.

Several papers supply methods or competing designs. SAS uses per-slot attention over history and K-means over slot representations, but only on clean MultiWOZ text. Learning to Relate to Previous Turns in Conversational Search learns turn relevance from pseudo-labels based on how much each turn helps retrieval. Multi-Granularity Information Interaction for Incomplete Utterance Rewriting softly down-weights irrelevant turns. We treat the last two as the strongest turn-level competitors. Decker & Amblard show that on Friends transcripts, feature-enhanced TextTiling beats a BERT coherence model on the Pk segmentation error metric (46.44 vs 48.43). This limits how much we can expect off-the-shelf neural embeddings to segment casual dialogue well, and it provides the segmentation-only baseline.

Research Program

Encoder. We start from a sentence encoder and continue its pre-training on unlabeled spontaneous transcripts: Fisher-Switchboard and SpokenWOZ training-set speech-recognition output. This is second-stage domain-adaptive pre-training, as in Caragea et al.. The purpose is to address domain mismatch. An encoder trained on formal text places disfluent fragments far from everything else, so they get flagged as outliers because the embedding space is unfamiliar with them, not because they are off-topic.

Corpus-level topics. We embed every training turn, reduce the embeddings with UMAP (a nonlinear projection that preserves nearest-neighbour structure), and cluster them with HDBSCAN (density-based clustering that leaves low-density points unassigned as "noise"). The clusters are fit once on the training set, and test turns are assigned to existing clusters. This avoids re-clustering each dialogue, which the topic-modeling literature identifies as a bottleneck for streaming conversation. The resulting clusters should correspond to recurring functional topics, such as "hotel constraints" or "number exchange," rather than to individual dialogues.

Episodes. Within a dialogue, an episode is a maximal run of turns sharing a cluster label. A turn labelled noise is merged into the episode of its nearest non-noise neighbour in time. This merging is the key ingredient, and the "absorb vs delete" ablation tests it directly.

History construction. At turn t, the tracker receives three things: the current episode verbatim, any earlier episode whose cluster matches the current turn's cluster verbatim (to handle returns to a topic), and every other earlier episode compressed to the tracker's own previous slot predictions from that episode.

Tracker. We use the WavLM-plus-LLM tracker unchanged, with a fixed tuning budget. We run it primarily at OLMo-1B scale, with a confirmatory run at 9B.

Baselines

The do-nothing reference is full-history concatenation into the same WavLM-plus-LLM tracker, retrained by us under identical data, steps and LoRA configuration. Published anchors are 34.66% JGA for OLMo-1B with fuzzy matching and 42.17% for Gemma-2-9B on the SpokenWOZ test set (Approaching DST via Aligning Speech Encoders and LLMs). Beating it means a statistically reliable JGA gain over our own reproduction at matched scale, concentrated on cross-turn slots. Lower-bound references are SPACE+WavLM at 25.65% and the cascaded Whisper system at 30.74%.

Two simple truncation baselines are a last-k-turns window, with k tuned on the development set, and "previous predicted state plus current turn." These are the cheapest ways to remove dilution. They are run in the same tracker with the same tuning budget as ours.

The strongest turn-level competitor is a learned turn selector trained with pseudo-labels, following Mo et al.. A turn is labelled relevant if removing it changes the tracker's predicted state. We run it in two variants, hard selection and soft masking in the style of Du et al.. It gets the same encoder and the same development-set tuning as our method. It is the strongest competitor because it is supervised by the task itself, whereas our method is not. Beating it requires either higher cross-turn-slot accuracy, or equal in-domain accuracy together with better held-out-domain accuracy.

The segmentation-only baseline replaces clusters with per-dialogue TextTiling-style boundaries, following Decker & Amblard. It isolates whether corpus-level clusters, which are what allow topic returns to be detected, add anything beyond contiguity.

Two ablations isolate our ingredients. One deletes HDBSCAN noise turns instead of absorbing them. The other uses a generic encoder instead of the adapted one.

Why not just…?

Why not just trust a large LLM with full history? This is a real threat. The 9B full-history model already reaches 42.17%, and SpokenWOZ dialogues may not be long enough for attention dilution to matter much. We settle it by binning dialogues by length and reporting per-bin JGA. If our method shows no advantage even in the longest bin at 9B scale, the dilution premise fails for this benchmark, and we will report that.

Why not just use a learned turn selector? It is task-supervised and may match us in-domain. Our bet rests on the granularity trade-off: a selector scores turns independently, so it can keep the first digits of a phone number and drop a later fragment that looks uninformative on its own. The cross-turn-slot breakdown and the held-out-domain split decide this. If the selector ties on both, topic structure is unnecessary and we would recommend the selector.

Why not just keep the previous state plus the current turn? That discards partial values that are not yet slot values, which is exactly what cross-turn slots consist of. The SpokenWOZ drop from 28.15% to 25.65% when cross-turn slots are included suggests these partial values matter, and the per-slot-type comparison will show whether state-only history fails there.

Proposed Experiments

The main evaluation is the SpokenWOZ test set. We report overall JGA, JGA on cross-turn and reasoning slots, input tokens per turn, and latency. For domain shift, we run a leave-one-domain-out split over SpokenWOZ's eight domains and a transfer test to MultiWOZ. For MultiWOZ, no reported result for the full-history speech-LLM tracker was found on Lacuna, so we will establish that baseline ourselves. To probe spontaneous-speech robustness, we feed histories transcribed by recognizers of varying quality and check whether the fraction of turns labelled noise, and the gap between absorbing and deleting them, grows as recognition errors increase. For the adapted encoder, we also measure how much it lowers the noise fraction relative to the generic encoder.

Three results would decide the question. The method must beat both full history and the learned selector on cross-turn slots. Deleting noise turns must hurt cross-turn slots specifically, while absorbing them must not. The margin over full history must grow with dialogue length and on held-out domains. If any of the three fails, the corresponding part of the claim does not hold.