TL;DR
This paper tackles the problem of spoken dialogue systems (SDSs) incorrectly responding to user utterances that aren't directed at the system, like mutters or back-channels. They propose a logistic regression model using dialogue-level features (like utterance timing and dialogue status) that significantly improves the accuracy of detecting system-directed utterances, even when Automatic Speech Recognition (ASR) errors are present.
What Problem Does This Paper Address?
Spoken Dialogue Systems (SDSs) are designed to interact with users through voice. A major challenge is distinguishing between user utterances that are intended for the system (e.g., questions, requests) and those that are not (e.g., a user muttering to themselves, a sigh, or a back-channel response like "uh-huh"). When an SDS incorrectly interprets a non-directed utterance as a command or question, it can lead to irrelevant or annoying responses, degrading the user experience. This is particularly problematic for new users who might be more prone to unintentional vocalizations. Existing methods often focus on ASR accuracy or simple acoustic thresholds, but this paper argues that dialogue context and timing are crucial for robustly identifying directed utterances.
Methods
The core of the proposed method is a logistic regression classifier. The authors introduce a novel set of features that go beyond just the content of the utterance (which relies heavily on ASR accuracy). These features are categorized as:
- Utterance Length: Longer utterances are hypothesized to be more intentional.
- Relationship with Preceding Utterance: This includes features like the time interval between utterances and whether the current utterance is a successive user utterance or a barge-in (user speaking during system utterance).
- Dialogue Status: Crucially, this feature indicates whether the system has "yielded the turn" to the user (e.g., by asking a question) or is still maintaining the turn.
- Utterance Content: Features derived from keywords indicating responses, requests, terminations, fillers, and content words relevant to the dialogue domain.
- Features from ASR Results: A verification score from the ASR system.
Importantly, the authors emphasize that features not dependent on ASR accuracy (like dialogue status and timing) are particularly valuable for robustness. They also employ backward stepwise feature selection to identify the most effective features for classification.
Datasets
The study used data collected from a Japanese spoken dialogue system designed to explain and answer questions about world heritages.
- Target Data: 5395 user utterances were collected. These were manually transcribed and annotated with semantic tags.
- Preprocessing: Utterances were merged based on short silences and semantic tags to form single units. This resulted in 5193 transcribed utterances (4257 directed, 936 not-directed) for manual transcription evaluation.
- ASR Data: Successive ASR results with short intervals were merged, yielding 4298 utterances (4096 directed, 202 not-directed) for ASR evaluation. Reference labels were assigned based on correspondence with the transcribed utterances.
No new benchmark datasets were introduced; rather, existing dialogue system data was utilized and annotated.
Key Insights and Ideas
- Dialogue Context is King: The most significant insight is that features related to the dialogue's state and timing are highly effective, even outperforming utterance content features when ASR is imperfect. This suggests that understanding when and in what context a user speaks is as important as what they say.
- Robustness to ASR Errors: The paper demonstrates that SDS-specific features (like dialogue status) are crucial for maintaining classification accuracy even when ASR results are noisy or erroneous. This is a major step towards building more forgiving and user-friendly SDSs.
- Barge-in Nuances: Differentiating between normal barge-ins and those that might be unintentional mutters is addressed through specific timing and relationship features.
- Feature Selection Matters: The use of backward stepwise feature selection helped identify and remove less effective features, leading to improved classification accuracy.
Key Results
- Transcription Classification: Using all features (including SDS-specific ones), the classification accuracy reached 85.1%. This improved by 11.0 percentage points compared to using only utterance length (74.4%). Removing SDS-specific features dropped accuracy to 84.2%.
- ASR Results Classification: Classification accuracy for ASR results was 76.7% after feature selection. This was a significant improvement over using only utterance length (72.6%). The SDS-specific features were shown to be effective, with a statistically significant improvement over a baseline without them.
- Feature Effectiveness: The SDS-specific features (relationship with preceding utterance and dialogue status) were found to be effective, contributing to the improved accuracy, especially in cases with ASR errors.
Limitations and Future Work
- ASR Error Impact: While the SDS-specific features improve robustness, the paper acknowledges that ASR errors can still significantly degrade classification accuracy, particularly when they affect content-based features (like keywords).
- Data Specificity: The study was conducted on a Japanese dialogue system explaining world heritages. The effectiveness of domain-specific content words might vary across different domains.
- Future Work: The authors suggest that further improvements could be made by incorporating more sophisticated dialogue state tracking and potentially exploring other machine learning models. The paper also implicitly points to the need for better end-point detection for utterances, as mentioned in footnote 1.
Summary updated