Discrete Dialogue-Act Codes Learned by Predicting the Next Turn

Conversations are built from a small set of communicative moves: asking for a phone number, confirming a booking, thanking someone. Two turns can make the same move with no words in common ("Count me in" and "I'd be happy to attend"). Two turns can also share almost every word and make different moves ("What is your phone number?" and "What is your name?"). Today, the best way to embed turns by their function is to pre-train on millions of turns hand-labeled with dialogue acts and slots, then run k-means on the resulting vectors to recover the steps of a conversation's workflow. That works where labels exist. It gives no principled route for a new domain, such as a hospital hotline or a support desk, where only raw logs are available.

We propose to learn the moves directly from unlabeled logs. An encoder compresses each turn through a small discrete bottleneck, a codebook of $K$ learned vectors from which each turn picks exactly one. Training does not reward the code for reconstructing the turn's own words. It rewards the code, together with the codes of the preceding few turns, for identifying which reply actually came next. The code only pays off if it carries what shapes the other speaker's response, so it should keep the communicative function and discard wording. Because the codes are discrete and learned jointly with the encoder, the resulting inventory of acts and the transition graph between them come straight out of training, with no separate clustering step.

Core Claim

A discrete bottleneck trained to predict the next turn contrastively can recover dialogue acts and conversation workflows from unlabeled logs. The bar is to clearly beat unsupervised clustering of generic sentence embeddings and to close a substantial part of the gap to action-supervised embeddings on domains those embeddings were never trained on. The specific ingredient under test is the joint discrete bottleneck. The same next-turn objective with a continuous output that is clustered afterwards is the ablation that decides whether the bottleneck matters.

The mechanism is easy to state. Let $e_t$ be a frozen general-purpose sentence embedding of turn $t$, and let the encoder map $e_t$ plus a speaker-role flag to the nearest of $K$ codebook vectors, $q_t$. A small predictor $h$ reads the code window $q_{t-w},\dots,q_t$ and is scored against the true next-turn embedding $e_{t+1}$ and in-batch negatives $e_j$:

$$\mathcal{L} = -\log \frac{\exp(\mathrm{sim}(h(q_{t-w:t}), e_{t+1})/\tau)}{\sum_j \exp(\mathrm{sim}(h(q_{t-w:t}), e_j)/\tau)} + \beta\,\mathcal{L}_{\text{commit}}.$$

Here $\mathrm{sim}$ is cosine similarity, $\tau$ a temperature, $w$ the context window, and $\mathcal{L}_{\text{commit}}$ the usual term that keeps encoder outputs close to their assigned codes. Gradients pass through the discrete choice with a straight-through estimator. The fixed-$K$ problem is handled by starting with a generous $K$ and pruning codes that stop being used. The act inventory's size is then an output of training rather than an input.

Where This Sits in Prior Work

The motivating evidence comes from action-supervised embedding. Dialog2Flow trains sentence embeddings with a soft contrastive loss over 3.4 million utterances labeled with 18 acts and 524 slots, then clusters them into workflow nodes. It reports 70.94 five-shot action F1, against 33.30 for Sentence-BERT. On SpokenWOZ it extracts workflow graphs within 8.5% of human references in complexity, where generic embeddings miss about half the steps. This shows that function-organized spaces matter a great deal. We drop its label requirement and its two-stage pipeline. Its clustering is post hoc; ours is the training objective.

Two unsupervised lines induce functional structure with latent variables. Unsupervised Dialogue Act Induction using Gaussian Mixtures fits a hidden Markov model with Gaussian emissions over IDF-weighted GloVe turn vectors. It reaches 65.7% F1 and 41.2% v-measure on Switchboard-DAMSL, against 74.5% supervised, but fixes $K$ and fits its emissions to static word vectors. Dialogue State Induction Using Neural Latent Variable Models uses a Gaussian-mixture variational autoencoder to induce slot values, reaching 49.6 turn-level F1 but only 5.0% joint goal accuracy on MultiWOZ 2.1. We target acts rather than slots, and we replace generative likelihoods with a discriminative next-turn objective.

Two papers supply the method and one constrains it. Learning Compressed Sentence Representations for On-Device Text Processing shows that a straight-through discrete bottleneck over sentence embeddings can retain downstream accuracy within about 2%. We reuse the estimator. Its target, however, is preserving the original cosine geometry, which would inherit exactly the lexical similarities we want to remove.

The constraint is Contextualized and Generalized Sentence Representations by Contrastive Self-Supervised Learning. It finds that predicting a masked sentence from its discourse context contrastively beats word-level generative reconstruction on implicit discourse relations: 63.30% versus 61.68% for base RoBERTa on PDTB 3.0, with the generative ablation lower still. This rules out a textbook reconstruction autoencoder, and it is why our "decoder" is a contrastive next-turn scorer. Does the Data Processing Inequality Reflect Practice? reminds us that compression cannot add information. Our claim is therefore about how the discarded information is chosen and how clusterable the result is, not about information gain.

Research Program

The build is deliberately small. The model is a frozen general sentence encoder, a two-layer encoder into a codebook starting at $K=256$, and a light transformer predictor over code windows. It is trained on the text of unlabeled dialogues. The first analysis asks what the codes keep. We probe whether a turn's own bag of words is more recoverable from the continuous embedding than from its code, and whether act labels are more recoverable from the code. The desired signature is that the code loses lexical content while retaining function.

The second analysis builds workflows. Code-to-code transition counts form a directed graph directly, and we compare it with the graph from clustering Dialog2Flow vectors. The third analysis tests stability under speech. Using SpokenWOZ's ASR transcripts, we measure how often turns that annotators assign the same act receive the same code. This asks whether the bottleneck filters disfluencies or instead destroys information delivered incrementally across turns, which is a real risk given that SpokenWOZ's cross-turn slots are a known failure point. The best reported joint goal accuracy there, 25.65% for SPACE+WavLM per SpokenWOZ, rises only to 28.15% when those slots are removed. Slot tracking itself is out of scope.

Baselines

The naive reference point is k-means over frozen Sentence-BERT turn embeddings, with and without PCA to 64 dimensions. It uses the same frozen encoder our model sits on, and it gets the same selection of $K$: either the value our pruning ends at, or the best $K$ from the same held-out sweep. On the Unified TOD five-shot action task, Sentence-BERT is reported at 33.30 F1 by Dialog2Flow. Anything that does not clearly beat this has learned nothing about function.

The strongest unsupervised prior method is the context-dependent Gaussian-mixture hidden Markov model of Brychcín and Král. It is reported at 65.7% F1 and 41.2% v-measure on Switchboard-DAMSL. We rerun it twice: once with its original GloVe inputs to reproduce that number, and once with the same frozen sentence embeddings we use, so that encoder quality is not confounded with method. Beating the reported numbers means exceeding both values under the same 42-act evaluation mapping. Beating the modern-input rerun is the fairer bar.

The strongest existing functional embedding is Dialog2Flow with k-means, as its paper does it. It serves as a supervised reference, not a competitor to beat everywhere. On datasets inside its pre-training mixture we expect it to win. The fair comparison is leave-dataset-out: domains held out from our training data and absent or removed from its labels as far as its released data allows. There, success means recovering at least half of the gap between Sentence-BERT and Dialog2Flow on five-shot action F1 and on workflow-graph fidelity, measured on the same unlabeled logs.

The key ablation keeps everything identical, including the next-turn contrastive objective, encoder, data, and tuning budget, but removes the discrete codebook and runs k-means on the continuous predictor inputs afterward. This isolates joint discretization. A second ablation swaps the next-turn target for reconstruction of the turn's own embedding, in the style of AE-SP. It tests whether the context target is what pushes codes toward function.

Why not just…?

Why not just cluster Sentence-BERT or PCA-reduced embeddings? Generic embeddings group turns by wording. The five-shot gap in Dialog2Flow (33.30 versus 70.94) and the roughly 50% of workflow steps that generic embeddings miss on SpokenWOZ both indicate that clustering cannot recover function the space never encoded. PCA only reorients that same space. The naive baseline tests this directly, and if it matches us, the proposal fails.

Why not just train the next-turn contrastive model and cluster afterwards? This is the real threat. Most of the benefit may come from the context objective rather than from the bottleneck, which is what Kiyomaru and Kurohashi's result would suggest. The continuous-then-cluster ablation settles it. If it ties the discrete model on act recovery and graph fidelity, we will report that the bottleneck's value is limited to direct graph extraction and compactness, and we will reframe the contribution around the objective alone.

Why not just use Dialog2Flow, or have an LLM label acts? Dialog2Flow needs millions of labeled turns and its authors flag unseen domains as open. The leave-dataset-out evaluation measures how far it transfers. LLM labeling needs an act taxonomy to be supplied, which is the dependence we want to remove. The one relevant number, zero-shot ChatGPT at 13.75% joint goal accuracy on SpokenWOZ, is about slots rather than acts and suggests fragility under ASR noise. We include GPT-4-style act labeling with the Switchboard taxonomy as a reference on Switchboard only, and treat it as taxonomy-dependent.

Proposed Experiments

The main unsupervised test is Switchboard-DAMSL. We train on its unlabeled conversations and map codes to the 42 acts by majority assignment, the protocol of Brychcín and Král, reporting F1 and v-measure. The claim holds if we beat their 65.7 / 41.2 and the modern-input rerun of their model.

The domain-transfer test uses the Unified TOD corpus from Dialog2Flow. We hold out whole datasets, train our model on their raw turns, and report five-shot action F1 against the three reference points: 33.30 Sentence-BERT, 42.21 DSE, and 70.94 Dialog2Flow. For held-out domains specifically, no reported result was found on Lacuna, so all methods are rerun there.

The spoken-noise test is workflow extraction on SpokenWOZ ASR transcripts, compared against human reference graphs with the complexity measure used by Dialog2Flow. We also report within-act code consistency, and separately, act-recovery accuracy on turns that carry cross-turn slot fragments.

The decisive check is the pair of ablations. The discrete model must beat continuous-then-cluster, and the next-turn target must beat self-reconstruction, on at least the Switchboard and held-out Unified TOD comparisons. It must also do so with codes whose bag-of-words probe accuracy is lower than that of the input embeddings. If either ablation matches the full model, the corresponding ingredient is not doing the work we claim.