TL;DR
This paper investigates how users of a real-world spoken dialog system (SDS), "Let's Go," adapt their vocabulary to match the system's. They found strong evidence that users do indeed entrain to system-introduced words, and this adaptation can be observed over time, suggesting that users need repeated exposure to fully adopt new terms.
What Problem Does This Paper Address?
Conversations between humans are dynamic, with speakers naturally adjusting their language to match each other – a phenomenon known as lexical entrainment. While previous studies, often using "Wizard of Oz" setups or simulated dialogues, have shown that users adapt to SDS vocabulary, there's a gap in understanding this behavior in real-world, deployed SDSs. Specifically, the authors wanted to know: 1) Do users of a live SDS actually entrain to its vocabulary? and 2) How does this entrainment manifest when the system introduces new words? This is crucial for designing more natural and effective SDSs, as entrainment can improve dialogue flow and task success.
Methods
The researchers employed two main methods:
- Linear Regression Analysis: They analyzed a large corpus of human-transcribed dialogues. For each "prime" word the system used, they measured the "distance" (number of turns) until the user used a related word. They then built a linear model to see if there was a correlation between this distance and the frequency of the user's response. This was done in both normal turn order and a randomized order to control for chance.
- Vocabulary Change Analysis: They systematically changed a subset of the system's vocabulary (from V1 to V2) in its prompts. They then compared the word frequencies in dialogues before the change (using V1) and after the change (using V2) to see if users shifted their word choices towards the new system primes. This analysis focused on the relative frequency of words.
Datasets
- Let's Go SDS Dialogs: The study utilized a year's worth of human-transcribed dialogues from the "Let's Go" bus information system. This dataset comprised 18,981 dialogues and 197,801 turns.
- V1 and V2 Vocabularies: The researchers defined two sets of vocabulary words. V1 represented the original system vocabulary, and V2 consisted of alternative words designed to have similar semantic content to V1 words.
Key Insights and Ideas
- Real Users Entrain: The most significant finding is that users of a deployed SDS do adapt their vocabulary to match the system's, and this is statistically significant.
- Entrainment is Gradual: While some adaptation might occur quickly, the study suggests that users need more exposure than a single call to fully entrain to new system primes. The proportion of V2 words continued to increase even after the second week of the vocabulary change.
- Word Frequency Matters: More frequent words in the system's vocabulary appear to be more easily entrained by users. Conversely, less frequent or domain-specific words (like "itinerary") showed weaker entrainment.
- Context is Key (Potentially): The study hints that the context in which primes are used might influence entrainment, with words facilitating interaction (like "request") showing strong adaptation.
Key Results
- Linear Regression: The linear regression analysis showed a weak but statistically significant correlation between the distance of a user's response from a system prime and the frequency of that response. The slopes indicated that user responses were more frequent when they occurred closer to the system's prime.
- Vocabulary Change: After changing system prompts to include V2 words, the frequency of these new words in user utterances increased significantly. For example, the word "preceding" was used 0 times in a year with V1 but 96 times in three weeks after being introduced as a V2 prime. The overall shift in word frequency towards V2 words was statistically significant (p=0.0011).
Limitations and Future Work
- Caller Identification: The study lacked caller identification, which prevented them from verifying if users were repeat callers and thus potentially more seasoned. This limitation makes it difficult to definitively confirm the hypothesis that novice users adapt more quickly than experts.
- Contextual Analysis: While the study identified what words were entrained, it did not deeply explore why certain words were more or less entrained, beyond general frequency and domain specificity. Future work will examine the specific contexts in which primes are used.
- More Complex Manipulations: The authors plan to explore more complex manipulations, such as using primes in contexts not directly related to their intended use, and developing SDSs with broader domains to facilitate such studies.
- Bidirectional Entrainment: The ultimate goal is to develop SDSs that not only detect user primes but also adapt their own prompts to use them, creating a truly bidirectional entrainment system.
Summary updated