Task-oriented dialogue systems track what the user wants by repeatedly updating a belief: a distribution over possible slot values such as price = cheap or area = north. Each turn's output from the language understanding model is treated as one more piece of evidence. Systems that hard-commit to the single most likely value are brittle. Systems that keep a calibrated distribution do much better. Yet in current trackers, every turn is trusted on a fixed scale. A turn where the speech recognizer garbled the audio counts the same as a crisp one, and a turn where the user changed their mind looks the same as a recognition error. We propose to give every turn a continuous trust weight. A weight of 1 means "use this turn's evidence fully" and a weight of 0 means "treat it as saying nothing." The weight is a function of cheap per-turn signals, and its parameters are learned by Expectation-Maximization (EM) from unlabeled dialogues. EM is an algorithm that alternates between guessing hidden variables and refitting parameters. Here the hidden variables are the user's true goal and whether each turn was reliable.
The trust weight is borrowed from evidential reasoning, also called Dempster-Shafer theory. In this view, a source can put belief not only on single answers but also on "it's one of these few," and on "I don't know." Shrinking a source's evidence toward "I don't know" is called discounting. Discounting is a soft blend between trusting and ignoring, the same move that speech front-ends make when they mix enhanced and raw audio instead of switching between them. The concrete bet is that three things can be learned without state labels: when to blend, how much to blend, and whether the remaining doubt is conflict (the user said two different things) or ignorance (the user never said anything usable). We also bet that the policy, which picks the system's next action, chooses better when it knows the difference between the two.
Core Claim
Let $m_t$ be the belief mass produced at turn $t$ for one slot. It is spread over single values, a small set of top candidates $C_t$, and the whole value set $\Theta$ ("don't know"). Discounting with trust $\alpha_t \in [0,1]$ gives
$$m_t^{\alpha}(A) = \alpha_t\, m_t(A) \ \text{ for } A \neq \Theta, \qquad m_t^{\alpha}(\Theta) = \alpha_t\, m_t(\Theta) + (1-\alpha_t).$$
We parameterize the trust as $\alpha_t = \sigma(w^\top f_t)$, where $\sigma$ is the logistic function and $f_t$ is a feature vector for the turn. The features are the speech recognizer's confidence, the tracker's knowledge uncertainty (mutual information across an ensemble), and the turn position. The discounted masses are combined across turns using a combination rule that sends conflict to an explicit conflict term rather than renormalizing it away. This avoids the known failure of Dempster's rule when sources nearly fully disagree. A goal-change variable $z_t$, with switching probability $\rho$, lets the true value jump between turns. The parameters $w$ and $\rho$ are fit by EM on unlabeled dialogues. The E-step runs forward-backward over each slot's value sequence, restricted to single values plus $C_t$ plus $\Theta$, so its cost grows linearly in the number of slot values. The M-step is a weighted logistic regression for $w$ and a count update for $\rho$.
What would be new is this: per-turn reliability for dialogue evidence is learned without labels, and the resulting belief keeps conflict separate from ignorance so that the policy can use the distinction. No paper in the file does either for dialogue.
There is one honest caveat, and it shapes the whole design. When the accumulated belief is an ordinary probability distribution $p$, combining it with a discounted source gives a posterior proportional to $p(s)\,(\alpha_t\,\mathrm{pl}_t(s) + 1-\alpha_t)$, where $\mathrm{pl}_t(s)$ is the source's plausibility of value $s$. This is exactly a Bayesian model in which each turn is, with probability $\alpha_t$, informative and otherwise noise. The evidential formulation therefore differs from a reliability-aware Bayesian tracker only through the mass it keeps on $C_t$ and $\Theta$. Our main claim is narrow on purpose: those extra quantities change policy decisions, especially the choice between request (when ignorance is high) and disambiguate (when conflict is high). They are not expected to raise tracking accuracy on their own.
Relation to Prior Work
Two dialogue papers motivate the gap. Spoken Dialogue Management Using Probabilistic Reasoning treats the user goal as hidden and compresses belief into most-likely-goal plus entropy, but its observation-error parameters were hand-set rather than learned from data. Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance distills an ensemble into SetSUMBT and shows that feeding confidence, entropy, and knowledge uncertainty to a reinforcement-learning policy raises success under noise. Its uncertainty is still a single-distribution summary, so ignorance and conflict are entangled. Knowing What You Know shows that calibrated ensembles place the true goal in the top three hypotheses far more often than baselines. That result motivates keeping a composite candidate set $C_t$ rather than collapsing to the single best value. We reuse SetSUMBT as the source of per-turn evidence and change only how turns are weighted and combined.
Three evidential papers supply the method and constrain its form. Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance separates confusion from ignorance with Subjective Logic in static image classification. the ICU fusion work fuses text and tabular evidence with Dempster's rule. T2MAC fuses agent messages evidentially and prunes uninformative messages. All three learn their evidence mappings with supervised gradients and use no learned, input-dependent discounting. All three also restrict the representation to singletons plus a few composite sets, and we adopt that same restriction for tractability. We differ by making reliability a latent quantity estimated by EM over time.
The trust-weight idea comes from speech front-ends. Learning to Enhance or Not replaces hard switching between enhanced and raw audio with a probability-weighted blend. It reports a gap to "oracle soft-switching," where the weight is chosen per utterance. We build the dialogue analogue of that oracle as an upper bound. Accurate endpointing with expected pause duration shows that expectations over the full recognition beam beat one-best decisions. It motivates using beam-level recognizer confidence as a trust feature, a signal that today never reaches the tracker.
The estimation side follows End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems, which argues that exact EM recovers switching regimes better than variational approximations. It also warns that identifiability requires assumptions that break under misspecification, which constrains our design. Estimating Prerequisite Structure From Noisy Data shows EM separating hidden states from slip and guess noise parameters. That is structurally the same problem as separating a user's goal changes from recognition errors.
Research Program
We first build the tracker: SetSUMBT turn-level outputs become per-slot masses. Single values get the student model's probabilities scaled by $1-u_t$, where $u_t$ is its knowledge uncertainty; $u_t$ itself goes to $\Theta$; and a fixed fraction of the remaining mass goes to $C_t$, the top-three set. We then fit $(w,\rho)$ by EM on MultiWOZ 2.1 training dialogues with the state labels hidden. We compare against a fit of the same parameters using labels, to measure how much supervision buys.
Next comes identifiability. A goal change and an unreliable turn can explain the same observed jump in the tracker's output, so we test whether EM separates the two. We do this with simulated dialogues in which goal changes and noise injections are planted with known rates. We expect separation to rely on the asymmetry between them: a goal change persists in later turns, while noise does not. If recovery fails, we will report the regime where it fails.
Finally, we train a PPO policy, as in the SetSUMBT paper, whose input adds belief, plausibility, conflict mass, and ignorance mass per slot. We then study the behaviour that produces any success gains.
Baselines
The naive reference is SetSUMBT with argmax states and a hand-set confidence threshold that triggers confirmation. It represents hard decisions at every interface.
The strongest existing system is the uncertainty-aware SetSUMBT plus PPO policy of Uncertainty Measures in Neural Belief Tracking. It reports 85.17% dialogue success under simulated noise (versus 78.67% for its baseline policy), a joint goal accuracy of 52.26%, and calibration error of about 7% on MultiWOZ 2.1. It is the strongest competitor because it already gives the policy calibrated uncertainty, and it does so cheaply. We will rerun it with the same ensemble, the same noise simulator, the same number of PPO training dialogues, and the same hyperparameter search budget as our system, and quote the reported figures only as a sanity check of our reproduction. As a calibration-focused tracker we also include CE-BST from Knowing What You Know, which reports 48.41% joint goal accuracy, 0.841 joint calibration error, and 84.08% top-three accuracy.
The key ablation is a reliability-aware Bayesian tracker. It uses the same features, the same goal-change variable, and the same EM, but its turns are a mixture of "informative" and "uniform noise," with no conflict or ignorance mass. By the equivalence noted above, this ablation isolates exactly what the evidential representation adds. Two further ablations isolate the learned weight: plain conflict-aware combination with $\alpha_t = 1$, and a single global $\alpha$ tuned on development data. The upper bound is an oracle per-turn $\alpha_t$, chosen with labels to maximize per-turn state accuracy.
Beating the strongest baseline means one of two outcomes on the noisy evaluation, with matched seeds and confidence intervals over at least five runs. The first is higher success at equal or fewer turns. The second is equal success with clearly fewer turns spent on confirmation. If the Bayesian ablation matches us on both, the contribution reduces to EM-learned reliability, and we will say so.
Why not just…?
Why not just use the distilled ensemble's uncertainty as the SetSUMBT paper already does? It already captures knowledge uncertainty per turn. However, it reports nothing about how evidence accumulates across turns, and a later confident turn overwrites an earlier one. Its policy also cannot tell "never mentioned" apart from "mentioned twice, inconsistently." The goal-change and noise splits of our evaluation directly test whether that distinction matters. If it does not, this baseline wins.
Why not just use a Bayesian tracker with a noise mixture, which is mathematically equivalent? This is the real threat. The two are equivalent for the posterior over single values whenever the accumulated belief is fully probabilistic. Our ablation is built to settle the question. If the conflict and ignorance features give no policy gain over the Bayesian ablation's entropy and mixture weights, we drop the evidential layer.
Why not just learn the trust weight with supervised gradients on labeled MultiWOZ? On in-domain data with labels, this probably works as well. EM matters only when labels are absent, for example when the noise conditions shift after deployment. The adaptation experiment below measures exactly that. We report the supervised fit as a peer, not a strawman.
Why not just drop low-confidence turns? That is hard thresholding. The front-end results in Learning to Enhance or Not, which report up to a 23% relative character error reduction from soft over rule-based switching, suggest that discarding partial evidence costs accuracy. We include a tuned drop-threshold variant to check whether the same holds for dialogue turns.
Proposed Experiments
All experiments use MultiWOZ 2.1 (corrections paper; MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines). Recognition noise is simulated as in the SetSUMBT evaluation, with an added controlled injection of goal changes. Policy training uses the agenda-based user simulator style of PyDial. The file contains no spoken version of MultiWOZ with real recognition lattices, so we found no reported result on Lacuna for real-ASR MultiWOZ tracking. Our recognizer-confidence feature will therefore come from simulation, and that is a stated limitation.
We measure tracking with joint goal accuracy, top-three accuracy, and calibration error. We measure the policy with success, turns per dialogue, and the confirmation rate. We add one diagnostic: on planted goal-change turns versus planted noise turns, does mass move to conflict or to ignorance respectively, scored by AUROC?
Three checks are decisive. The first is the adaptation test. Fit $w$ on clean dialogues, then run EM on unlabeled dialogues at a higher noise level, and compare against the frozen fit and the supervised fit. EM should close most of the gap to the oracle $\alpha_t$ without labels; if it does not, the EM component fails. The second is the evidential test against the Bayesian ablation, judged on policy success and confirmation turns under mixed goal-change and noise conditions. The third is the planted-recovery test: whether the learned $\rho$ and the learned reliabilities match the injected rates. A failure here would show that, in this setting, goal changes and recognition errors cannot be separated without labels. That would be a useful negative result in its own right.
References
- Spoken Dialogue Management Using Probabilistic Reasoning
- Uncertainty Measures in Neural Belief Tracking and the Effects on Dialogue Policy Performance
- Knowing What You Know: Calibrating Dialogue Belief State Distributions via Ensembles
- Flexible Visual Recognition by Evidential Modeling of Confusion and Ignorance
- Towards Accurate and Reliable ICU Outcome Prediction: A Multimodal Learning Framework Based on Belief Function Theory using Structured EHRs and Free-Text Notes
- T2MAC: Targeted and Trusted Multi-Agent Communication through Selective Engagement and Evidence-Driven Integration
- Learning to Enhance or Not: Neural Network-Based Switching of Enhanced and Observed Signals for Overlapping Speech Recognition
- Accurate endpointing with expected pause duration
- End-to-End Identifiable and Consistent Recurrent Switching Dynamical Systems
- Estimating Prerequisite Structure From Noisy Data
- MultiWOZ 2.1: Multi-Domain Dialogue State Corrections and State Tracking Baselines
- PyDial: A Multi-domain Statistical Dialogue System Toolkit