TL;DR
This paper introduces the Finite-State Turn-Taking Machine (FSTTM), a novel probabilistic model for controlling conversational agent turn-taking. By combining a finite-state machine with decision theory and a cost matrix, the FSTTM offers a flexible and data-driven approach to end-of-turn detection, significantly improving system responsiveness in spoken dialog systems.
What Problem Does This Paper Address?
Conversational turn-taking, the natural back-and-forth of speech and silence, is crucial for smooth human-computer interaction. Existing spoken dialog systems often rely on simplistic, ad-hoc rules for turn-taking, leading to unnatural and inefficient conversations. While more complex models exist, they are often hand-coded and not deployed in real-world systems. This paper tackles the challenge of creating a practical, flexible, and data-driven model for turn-taking that can be integrated into deployed systems, specifically focusing on the critical task of end-of-turn detection.
Methods
The core of the proposed solution is the Finite-State Turn-Taking Machine (FSTTM). This model extends a classic six-state finite-state machine (inspired by Jaffe & Feldstein, 1970) to represent the conversational floor. The states capture who holds the floor (System, User), if the floor is free (FREEs, FREEu), or if both participants claim it (BOTHS, BOTHu).
What makes the FSTTM novel is its integration of decision theory and a cost matrix. Instead of deterministic rules, the system makes turn-taking decisions by selecting the action with the lowest expected cost. This cost is calculated based on the system's belief about the current state (derived from observable features) and a predefined cost matrix that quantifies the penalty for undesirable events like gaps or overlaps. The actions available to the system are: Keep the floor (K), Release the floor (R), Wait without the floor (W), and Grab the floor (G).
The paper focuses on applying this framework to end-of-turn detection. This involves estimating the probability that the user has released the floor, P(F|Ot), and using this to decide whether to wait for a pause or grab the floor. They propose separate methods for estimating these probabilities when a pause is detected by a Voice Activity Detector (VAD) and when the system needs to make a decision "in speech" based on partial hypotheses from the Automatic Speech Recognizer (ASR).
Datasets
The authors evaluated their FSTTM on two datasets:
- "2007 corpus": This corpus was used for comparing the in-pause FSTTM with previous data-driven endpointing methods.
- "2008 corpus": A more recent corpus used for evaluating the anytime-FSTTM against in-pause FSTTM and a fixed-threshold baseline. This corpus was collected from a deployed bus schedule information system.
No new benchmarks were introduced; instead, the evaluation focused on comparing the FSTTM against existing methods and baselines.
Key Insights and Ideas
- Probabilistic Turn-Taking is Key: Moving beyond deterministic rules to a probabilistic, decision-theoretic approach allows for more robust and adaptable turn-taking behavior, especially in the face of uncertainty.
- Cost Matrix as a Formalization of Interaction Goals: The cost matrix elegantly captures the human preference for minimizing gaps and overlaps, providing a principled way to define system actions.
- Boundary LM Score is Highly Informative: For end-of-turn detection, a feature derived from statistical language models estimating the likelihood of a hypothesis being complete (the "boundary LM score") proved to be the most significant predictor of turn finality. This highlights the value of lexical cues.
- "Anytime" Endpointing is Superior: The ability to make endpointing decisions even during speech, not just at pauses, significantly reduces latency and improves responsiveness.
- Flexibility for Data-Driven Optimization: The FSTTM's structure is inherently amenable to data-driven optimization, allowing costs and probabilities to be learned from real conversational data.
Key Results
- Batch Evaluation: The FSTTM outperformed previous data-driven endpointing methods and fixed-threshold baselines, improving latency by up to 29.5% on the 2007 corpus. On the 2008 corpus, the anytime-FSTTM yielded latencies that were on average 17% shorter than in-pause-FSTTM and 40% shorter than the baseline, at a comparable cut-in rate.
- Live Evaluation: In a live deployment, the FSTTM reduced average latency by 193 ms (p < 0.05) and the cut-in rate by 1.5% compared to a fixed-threshold baseline, demonstrating its effectiveness in a real-world scenario.
- Feature Importance: The boundary LM score was consistently the most informative feature for estimating turn finality across different dialog states.
Limitations and Future Work
- Simplifying Assumptions: The paper acknowledges simplifying assumptions, such as the assumption that only USER and FREEu states are probable once a user utterance is detected. It also notes that the cost structure is a first approximation and could be refined through user studies.
- Observable State Uncertainty: The system's knowledge of the user's floor claim is not perfectly observable, requiring probability estimation.
- Future Work: The authors suggest exploring more principled approaches like Partially Observable Markov Decision Processes (POMDPs) or Dynamic Bayesian Networks to better model uncertainty. They also propose extending the state structures for multi-party conversations and developing data-driven cost matrices.
Summary updated