Sparse Variational Slot Induction for Human-Reviewed Dialogue Schemas

Before a task-oriented dialogue system can be built, someone has to write down its schema: the list of "slots" (price range, arrival time, has-parking) the system must collect from users and pass to a database. Recent systems try to recover that list automatically from raw conversation logs. They do it by having a language model propose candidate slot–value pairs and then clustering those proposals. The results are badly bloated. The strongest generative method induces 180 slots on MultiWOZ where human designers used 31, and many of the extra "slots" are greetings or clarification requests rather than database fields (Transforming Slot Schema Induction with Generative Dialogue State Inference). In practice a developer must then clean up that list by hand. That is the step older schema-compilation platforms called "semi-automatic," and it is where most of the labor ends up.

We propose replacing the final clustering step with a small Bayesian model fit by variational inference, which turns posterior estimation into an optimization problem. In this model each candidate slot must explain three things together: the embedding of its proposed name, the distribution of values it takes, and whether it looks like a real field at all rather than conversational filler. A sparsity prior over slot weights penalizes unnecessary slots. The posterior then serves two purposes. It yields a compact schema, and it supplies a ranked list of the merge, split, and discard decisions the model is least sure about, so a human reviewer spends a fixed budget where it matters most. The claim is that much of the overgeneration comes from the inference procedure rather than from the proposals themselves. A second claim is that calibrated uncertainty over slots makes human curation measurably cheaper than reviewing deterministic clusters.

Core Claim

Density clustering such as HDBSCAN groups candidates only by geometric proximity in embedding space. That causes two documented failures. First, near-synonymous names that sit at medium distance end up in separate clusters, which inflates the slot count. Second, slots with little lexical variety cannot form a dense region at all. Binary or single-value slots such as has_wifi=True are the case that Unsupervised Slot Schema Induction for Task-oriented Dialog explicitly reports as failing. A generative mixture with a categorical value likelihood treats a slot whose values are always "yes"/"no" as a perfectly good, highly probable explanation, rather than an empty region. A sparse prior makes merging two clusters that share a value distribution cheaper than keeping both. An explicit "not-a-slot" background component gives discourse acts somewhere to go. Each of these three ingredients can be removed in isolation, and the experiments below are designed to show which ones actually matter.

Model and Inference

We take the candidate triples produced by the GenDSI generator, unchanged. Each candidate $i$ is a proposed slot name with sentence-encoder embedding $e_i$, a normalized value string $v_i$, and the dialogue turn it came from. Each candidate has a latent assignment $c_i \in \{0, 1, \dots, K\}$. Here $K$ is a generous truncation level (say 400), and component $0$ is a background "discourse" component. The joint model is

$$ p(e, v, c, \pi, \mu, \phi) = p(\pi)\prod_k p(\mu_k)\,p(\phi_k)\prod_i \pi_{c_i}\,\mathcal{N}(e_i \mid \mu_{c_i}, \sigma^2 I)\,\mathrm{Cat}(v_i \mid \phi_{c_i}). $$

In this equation, $\pi$ are slot weights drawn from a truncated stick-breaking prior with a small concentration parameter, which drives most weights toward zero. $\mu_k$ is the mean name embedding of slot $k$. $\phi_k$ is its value distribution, under a Dirichlet prior over a shared value vocabulary. A separate open-vocabulary character model handles slots whose values rarely repeat, such as names and addresses. The background component additionally conditions on turn position and a lightweight dialogue-act cue, so that greetings and acknowledgments are pulled there.

Every factor is conjugate, so mean-field coordinate updates are closed-form. This is the same style of fixed-point iteration that Factorial Learning and the EM Algorithm found to converge in about a dozen steps and to perform comparably to Gibbs sampling. The sparsity mechanism is borrowed in spirit from Deep Exponential Families, where sparse Gamma priors let some latent factors absorb, or "explain away," what other factors would otherwise represent, which removes redundant factors. An optional second layer groups slots into domains. Black-box gradient estimators are not needed, which avoids their well-known variance problems.

There is a known risk. Mean-field posteriors underestimate variance, and our review ranking depends on that variance. We therefore check calibration of the merge and discard probabilities directly, and we compare the review ranking against a short Gibbs run on the same model.

The semi-automatic step works as follows. For each pair of surviving slots we compute the posterior probability that they should be one slot, and for each slot the probability that it belongs to the background. A simulated or real reviewer answers the $B$ highest-entropy questions. Each answer is clamped as an observation and inference is rerun, which is cheap. The final posterior is compiled into per-slot input collectors: categorical if $\phi_k$ is concentrated, open-span otherwise. These follow the atomic input/output modules that Strategies for accelerating the design of dialogue applications using heuristic information from the backend database generated from database fields. The difference is that our evidence comes from dialogue logs rather than SQL statistics.

Relation to Prior Work

Two schema-induction systems supply our inputs and our main competitors. Unsupervised Slot Schema Induction for Task-oriented Dialog extracts spans with a grammar-regularized span model and clusters them hierarchically with HDBSCAN. Transforming Slot Schema Induction with Generative Dialogue State Inference generates abstract slot names with T5-3B and clusters their embeddings. We keep GenDSI's proposals and change only the step that decides how many slots exist and which proposals belong together.

The two existing variational dialogue models address different problems. LAVA shapes a latent action space for a reinforcement-learning dialogue policy. Variational Hierarchical Dialog Autoencoder generates augmentation data from a fixed, densely annotated ontology. Neither induces a schema from unlabeled logs. Neither uses posterior uncertainty to direct a human. VHDA's need for gold belief states is exactly what we remove.

Human-in-the-loop Schema Induction motivates the review component. It found that GPT-3 produced accurate individual steps but needed heavy human correction of graph structure, which suggests that a model which tells the human where to look is valuable.

Baselines

The naive reference point is GenDSI's own output: its proposals clustered by HDBSCAN with the published settings. The file reports 180 induced slots against 31 gold slots and 93.5% accurate cluster names. It reports no slot-matching F1 for GenDSI on MultiWOZ ("no reported result found on Lacuna" for that figure), so we rerun it on the same candidate set.

The strongest published baseline on schema quality is TOD-Span with grammar regularization (Unsupervised Slot Schema Induction for Task-oriented Dialog). It reports a slot type F1 of 96.67, a slot value F1 of 58.71, and a downstream zero-shot joint goal accuracy of 46.69 on MultiWOZ. Beating it means matching its slot type F1 within noise while exceeding 58.71 on value F1, and producing a slot count much closer to 31 than GenDSI's 180, all under the same evaluation script. Slot count alone does not count as a win. A schema with fewer slots but lower recall is a loss.

A third baseline is HDBSCAN with its minimum-cluster-size and merge threshold tuned on the validation dialogues. It gets the same number of tuning trials as our two prior hyperparameters, so any gain from our model cannot be attributed to extra tuning.

Two ablations isolate the key ingredient. The first is a standard variational Dirichlet-process Gaussian mixture over name embeddings only. This keeps the sparse prior but drops the value likelihood and the background component. The second is our full model without the sparse prior, with uniform weights. All methods consume identical candidate triples and encoder embeddings. For the human-review comparison, every method gets the same number $B$ of reviewer answers. Deterministic baselines rank their questions by HDBSCAN membership probability or by random order.

Why not just…?

Why not just tune the clustering threshold? Overgeneration might be nothing more than a badly chosen minimum cluster size. That is a real possibility, and the tuned-HDBSCAN baseline exists to test it. Thresholding alone cannot fix the single-value-slot failure that the TOD-Span paper documents, because no threshold lets a density method see a region that has no variance. We report recall separately on MultiWOZ's yes/no slots, such as hotel internet and parking, to test this. If tuned HDBSCAN matches us there as well, the model-based part of our contribution disappears, and only the review-ranking result would remain.

Why not just ask an LLM to consolidate the 180 slots? This is the cheapest strong competitor. We run it as a baseline: a large model prompted with the cluster names and sample values, asked to merge clusters and discard non-fields. The evidence from Human-in-the-loop Schema Induction is that LLMs handle local items well but structural decisions poorly. An LLM pass also gives no calibrated signal for routing human attention. If LLM consolidation matches our F1, we test whether combining it with our posterior ranking still reduces reviewer effort. If it does not, we report that it does not.

Why not just use an off-the-shelf Bayesian mixture? The first ablation is exactly this. If it captures most of the gain, then the honest conclusion is that a sparse prior is all that is needed, and the value likelihood and background component are unnecessary additions. We would then simplify the method accordingly.

Proposed Experiments

Schema induction is evaluated on MultiWOZ 2.1, matching GenDSI's setting, and on the Schema-Guided Dialogue corpus (Schema-Guided Dialogue State Tracking Task at DSTC8), whose test set includes 15 services unseen in training. Gold schemas from MultiWOZ 2.2 supply the categorical versus open-span labels used to score our compiled collector types. Metrics are slot type and value precision, recall, and F1 under the TOD-Span matching protocol. We also report induced slot count, recall on single-value slots, the fraction of discourse-act clusters that are correctly sent to the background component, and expected calibration error of the merge and discard probabilities, judged against gold. Downstream, the induced schemas are used to train a zero-shot state tracker, as in TOD-Span, scored by joint goal accuracy with the official MultiWOZ code.

The decisive check is a review-budget curve: schema F1 against the number of oracle reviewer answers, for budgets from 0 to 100, compared against the HDBSCAN-probability ordering and random ordering. The proposal succeeds if our method needs substantially fewer answers to reach the F1 the baselines reach at full budget, and if its zero-review schema already beats TOD-Span's 58.71 value F1. A small live study, in which developers curate an SGD service schema with each tool, would then test whether the simulated gains hold when real people answer the questions.