TL;DR
This paper tackles the challenge of addressee detection (AD) in human-human-computer (HHC) dialogs, where systems need to distinguish between speech directed at the computer and speech directed at other humans. They demonstrate that using out-of-domain data, particularly from human-computer (HC) and human-human (HH) interactions, can significantly improve AD performance compared to using only limited in-domain HHC data, even outperforming matched in-domain training.
What Problem Does This Paper Address?
In multi-person dialog scenarios where a computer system is present, it's crucial for the system to identify which speech is intended for it. This is known as addressee detection (AD). Without accurate AD, systems might misinterpret human-to-human conversations as commands or requests, leading to errors and a poor user experience. Traditional AD research has focused on human-human (HH) settings, but the rise of HHC environments, like collaborative robots or smart assistants in group settings, necessitates AD for these more complex interactions. A key challenge is that collecting large amounts of in-domain HHC data is expensive and time-consuming, especially for new domains. This paper investigates whether readily available out-of-domain data can effectively substitute for or augment scarce in-domain HHC data for lexical AD.
Methods
The core of their approach is a lexical AD system based on language modeling. They train separate language models for "human-directed" (H) and "computer-directed" (C) utterances. The system then calculates a likelihood ratio between these models to determine the addressee.
Key methodological aspects include:
- Language Model Interpolation: To combine data from different sources (in-domain and out-of-domain), they use a weighted interpolation technique. This allows them to leverage multiple datasets while controlling their influence.
- Part-of-Speech (POS) Tagging: To address data sparsity and improve generalization, they replace less frequent words with their POS tags. This helps the model learn patterns from similar word classes, especially for unseen words. They retain the most frequent word types as distinct tokens to preserve domain-specific information.
- Online Processing Simulation: They investigate the feasibility of an online AD system by simulating performance using a "windowed" approach, analyzing how quickly AD decisions can be made as an utterance unfolds.
Datasets
The paper uses a combination of in-domain and out-of-domain datasets:
- In-domain Data: Collected from interactions between two users and a "Conversational Browser" (CB) spoken dialog system in a multi-human scenario (HHC).
- Out-of-domain Data:
- Single-user CB: Data from the same CB system but with only one user, used for modeling H-C speech.
- Bing anchor text: A large corpus of anchor text from web pages, hypothesized to help model "verbal clicks" in H-C utterances.
- Fisher corpus: Two-person telephone conversations (HH).
- ICSI meetings corpus: Multiparty face-to-face technical discussions (HH).
Key Insights and Ideas
- Out-of-Domain Data is Powerful: The most striking finding is that out-of-domain data alone can outperform a system trained solely on matched in-domain HHC data. This is a significant result for practical AD system development, as it suggests leveraging existing, readily available corpora.
- HHC AD Can Be Approximated: The paper concludes that the HHC scenario can be effectively approximated by combining data from H-C and H-H scenarios. This simplifies the data collection problem considerably.
- Early Utterance Information: Most of the lexical information for addressee detection is concentrated in the first 1.5 seconds of an utterance. This highlights the potential for efficient, early detection in online systems.
- POS Tagging for Sparsity: Replacing less frequent words with POS tags provides a significant performance boost, especially for the in-domain baseline model, which suffers most from data scarcity.
- Computer-Noncommand Utterances are Key: The biggest gains from using out-of-domain data are seen in detecting computer-directed noncommand utterances, which are naturally confusable with human-directed speech due to their unconstrained language.
Key Results
- Performance Gains:
- A system trained only on out-of-domain data (Fisher+ICSI, Single-user CB+Bing) achieved a 3.3 EER (Equal Error Rate) improvement over the in-domain baseline on ASR output.
- Interpolating in-domain and out-of-domain data further improved performance, with the "both-small" configuration (in-domain + smaller out-of-domain corpora) outperforming the "both-all" configuration.
- The best overall performance (lowest EER) was achieved by combining in-domain and out-of-domain data, reaching an EER of 13.0% on transcripts and 14.0% on ASR output.
- Online Detection: The windowed analysis showed that performance levels off significantly after 1.5 seconds, confirming the feasibility of online AD.
- POS Tagging Impact: POS-based models improved EER by 1.5 points (ASR) and 1.1 points (transcripts) over word-only models.
Limitations and Future Work
- ASR Mismatch: The paper notes a significant train-test mismatch for human-directed utterances when using ASR output, particularly when out-of-domain HH corpora (which are often transcripts) are used for training. This leads to poorer recognition of HH utterances in the ASR condition.
- Perplexity vs. Discrimination: While perplexity is a standard metric for optimizing interpolation weights, the authors found that search-based optimization using EER did not yield significant improvements, suggesting that perplexity might not be the ideal criterion for selecting data sources for discriminative tasks like AD.
- Future Work: The authors suggest further investigation into good criteria for corpus selection for classification tasks. They also implicitly point towards improving ASR robustness for human-directed speech in HHC scenarios. The feasibility of online AD is established, opening avenues for real-time system integration.
Summary updated