Apple Core-dination: Linguistic Feedback and Learning in a Speech-to-Action Shared World Game

Susann Boy, AriaRay Brown, Morgan WixtedACL 2021

TL;DR

This paper introduces "Apple Core-dination," a shared-world game designed to study how adaptive linguistic feedback from an AI agent influences user speech input. By analyzing user interactions and the agent's learning process, the research sheds light on how to build more effective human-AI communication systems.

What Problem Does This Paper Address?

Creating natural and effective communication between humans and AI agents, especially in interactive environments, remains a significant challenge. Specifically, how an AI agent provides feedback and learns from user input can drastically impact the user's own communication style and the overall success of the interaction. This paper investigates this by exploring how individualized linguistic feedback from a virtual agent in a shared-world game environment affects the user's instructional speech. The goal is to understand how to design agents that can adapt to and learn from human language in a way that fosters better collaboration.

Methods

The core of this research is the "Apple Core-dination" game, a speech-controlled shared-world environment. The game features an agent equipped with a learning mechanism that maps new commands to sequences of simple actions, drawing inspiration from semantic parsing models. The agent starts with a basic lexicon of actions and can incorporate user input to build its knowledge base.

Key methodological aspects include:

  • Speech-to-Text Flexibility: The system supports multiple speech-to-text (STT) models (Google Cloud Speech API and SpeechBrain), allowing for experimentation on user preference and transcription accuracy.
  • Adaptive Linguistic Feedback: The agent provides feedback through visual cues and written responses. Crucially, this feedback incorporates parts of the user's utterance, informing the user about the agent's understanding and knowledge state.
  • Learning Mechanism: The agent learns by mapping new commands to its existing knowledge base. When a complex task is successfully completed, the corresponding phrase-to-action mapping is added to the agent's learned phrases.
  • Data Collection: A pilot study was conducted with five volunteers. Transcripts of user-agent interactions were collected, including the user's instruction, the STT output, the agent's actions and response, and whether a task was completed. These transcripts were manually annotated for accuracy.

Datasets

The paper does not explicitly mention pre-existing datasets for training or evaluation. Instead, it relies on data collected from a pilot study involving five English-speaking volunteers playing the "Apple Core-dination" game. A total of 219 commands were gathered, with 182 annotated as "clean" transcriptions (where the STT model correctly transcribed the instruction, or the intended action was achieved despite minor errors).

Key Insights and Ideas

  • User Language Adapts to Agent Feedback: The study suggests that users tend to adjust their language based on the agent's feedback. When the STT model failed to transcribe correctly, users often repeated their commands, indicating a reliance on the agent's ability to understand.
  • Individualized Feedback Matters: The paper highlights that individualized linguistic feedback, which reflects the agent's understanding of the user's input, is crucial for successful communication. This feedback helps users understand the agent's knowledge and limitations.
  • Quantity vs. Quality of Input: Preliminary results show variation in both instruction length (quantity) and vocabulary size (quality) across users. The paper posits a potential correlation between these factors and suggests further investigation.
  • STT Model Choice Impacts Interaction: The choice of STT model can influence user interaction, with users showing preferences that might be due to usability, transcription accuracy, or speed.

Key Results

  • Average Instruction Length: Users tend to use short commands, with an average instruction length of 3.7 words.
  • Vocabulary Size: The average vocabulary size across users was 35.4 words.
  • Task Completion: The first task ("go to the tree") was easily completed by all users, serving as positive reinforcement. However, only one user completed all four tasks, suggesting potential challenges in task complexity or game engagement.
  • STT Model Preference: The SPACE bar (Google Speech API) was pressed more often than the M key (SpeechBrain), indicating a potential user preference for the Google model, possibly due to ease of use or better transcription.

Limitations and Future Work

  • Pilot Study Limitations: The study was a pilot with only five participants, limiting the generalizability of the findings. The authors acknowledge that the results are preliminary.
  • Task Complexity: The fact that only one user completed all tasks suggests that either the tasks are too difficult, or the game's engagement level needs improvement.
  • Future Work:
    • Conducting a larger comparative study with more participants, dividing them into groups receiving different types of feedback (linguistic vs. action-only).
  • Expanding the agent's baseline knowledge and improving its parsing abilities.
  • More finely incorporating user input into feedback responses.
  • Potentially using a semantic parser to generate more grammatically formed feedback for learned phrases.
  • Investigating the correlation between input quantity, quality, and agent feedback more precisely.
  • Encouraging users to try both STT models to better understand preferences and their impact.

Summary updated

BibTeX
@inproceedings{boy2021apple,
  author = {Susann Boy and AriaRay Brown and Morgan Wixted},
  title = {{Apple Core-dination: Linguistic Feedback and Learning in a Speech-to-Action Shared World Game}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure