Lexical and Syntactic Adaptation and Their Impact in Deployed Spoken Dialog Systems

Svetlana Stoyanchev, Amanda StentNAACL 2009

TL;DR

This paper investigates how users adapt their language to spoken dialog systems in real-world deployments, finding that users mirror the system's word choices and sentence structures. Crucially, this adaptation influences the system's ability to correctly understand user requests, suggesting that prompt design can significantly improve task success.

What Problem Does This Paper Address?

Spoken dialog systems (SDSs) often struggle with accuracy, especially "flexible input" systems that allow users more freedom in how they respond. While lab studies show people naturally adapt their language to their conversation partners (a phenomenon called "priming"), it's unclear if this holds true for real users interacting with live, deployed systems in noisy, everyday environments. This paper addresses this gap by examining user adaptation in CMU's "Let's Go!" bus information system, a deployed flexible input dialog system, and its impact on speech recognition accuracy.

Methods

The researchers analyzed dialogs from the "Let's Go!" system, which provides bus route information. They designed four experimental conditions, varying the lexical (word choice) and syntactic (sentence structure) formulation of system prompts for requesting and confirming departure and arrival locations. For instance, prompts might include verbs like "leave" or "go," or prepositions like "from" or "to," in different grammatical forms. They then analyzed user responses to these prompts, looking for evidence of adaptation in their word choices and sentence structures. Finally, they measured how these system prompt variations affected the accuracy of recognizing location concepts within user utterances.

Datasets

The study used dialog data from CMU's "Let's Go!" spoken dialog system. This system is a deployed, real-world application used by naive callers from the general population seeking bus information in Pittsburgh. The researchers collected 2184 dialogs across four experimental conditions over two months. No new benchmark datasets were introduced.

Key Insights and Ideas

  • Real-world adaptation is real: Users of deployed spoken dialog systems, just like in lab settings, adapt their language to match the system's. This means the system's phrasing influences how users speak.
  • Prompt design matters for accuracy: The way a system asks for information can directly impact its ability to understand the user. Specifically, certain prompt formulations led to significantly higher rates of correctly identifying user-specified locations.
  • Lexical priming is strong: When the system used specific verb forms (like "leaving" or "going"), users were more likely to adopt those same verb forms in their responses. This effect was particularly pronounced when the system used progressive verb forms.
  • Syntactic priming is also present: Users tended to use verbs and prepositions in their responses when the system prompts included them, suggesting a mirroring of grammatical structures.
  • Prepositional priming is subtle: While a trend was observed, the impact of system prepositions on user preposition use was not statistically significant, possibly due to the nature of prepositions as short, easily misrecognized words and the limited data.

Key Results

  • Syntactic Adaptation: Users were significantly more likely to use verbs in their responses when the system prompts contained verbs (conditions 1, 2, and 4) compared to when they did not (condition 3). Similarly, users showed adaptation in response to confirm location prompts containing verbs.
  • Lexical Adaptation: The system's choice of verb form had a statistically significant impact on user verb form choice. In the "progressive form" condition (1), users increased their use of progressive verbs by 13.2% (to leave) and 17.8% (to go). Conversely, in the "simple form" condition (4), users were more likely to choose simple verb forms.
  • Concept Identification Rates: Condition 4, which used the verb "to leave" in its request location prompt, achieved the highest concept identification rates for both arrival and departure locations. Differences between condition 4 and other conditions were statistically significant for request arrival location.

Limitations and Future Work

  • Data limitations: The authors note that the low relative frequency of dialogs in condition 4 might have obscured statistically significant differences in preposition use between conditions. They also suggest that more data or human transcription might be needed to confirm trends in prepositional priming.
  • Future Work: The researchers plan to confirm their findings using transcribed data. They also aim to conduct further experiments on adaptation in "Let's Go!", including analyzing the time course of adaptation and its impact on ASR performance. They also plan to explore additional experimental conditions.

Summary updated

BibTeX
@inproceedings{stoyanchev2009lexical,
  author = {Svetlana Stoyanchev and Amanda Stent},
  title = {{Lexical and Syntactic Adaptation and Their Impact in Deployed Spoken Dialog Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure