TL;DR
This paper introduces a novel multimodal corpus for training conversational agents to assist human operators in planning missions for autonomous underwater vehicles (AUVs). Their Wizard-of-Oz study highlights the critical role of combining natural language with visual plan representations for effective collaboration.
What Problem Does This Paper Address?
Planning missions for complex autonomous systems like AUVs is currently a challenging, expert-driven process, often relying on specialized visual interfaces. This makes it difficult for operators to quickly and intuitively communicate goals and collaboratively develop plans, especially in dynamic environments. The authors aim to bridge this gap by developing a conversational agent that can facilitate this planning process through natural language and visual feedback, reducing training time and enabling more flexible, in-situ replanning.
Methods
The core of this work is a Wizard-of-Oz (WoZ) study. In this setup, a human operator (the "subject") interacts with a simulated conversational agent. Behind the scenes, two human "wizards" act as the agent: one (the "Chatting Wizard") handles the natural language conversation, while the other (the "SeeTrack Wizard") translates the operator's requests into a plan using specialized AUV planning software (SeeTrack) and provides visual updates. This allows the researchers to collect realistic interaction data without needing a fully developed AI agent. The WoZ interface was designed to support multimodal interaction, allowing the Chatting Wizard to send text messages and images of the plan. They also analyzed the collected dialogs using a predefined set of dialog acts, categorizing them into generic, inform, request, suggest, and image.
Datasets
The paper introduces a new dataset: a corpus of 22 dialogs collected from expert AUV operators planning missions. These dialogs were gathered during a Wizard-of-Oz study involving two types of missions (surveying areas and target reacquisition) with varying complexities. The dataset is available upon request.
Key Insights and Ideas
- Multimodality is Crucial: The study strongly suggests that combining natural language with visual representations of the plan is essential for successful collaborative planning. Simply relying on text is insufficient.
- Spatial Referencing is Key: In the context of AUV mission planning, users frequently use spatial references (e.g., "move target2 200m west of target1"). The system needs to be able to understand and ground these references within the visual plan.
- User Experience Matters: Subjects found the multimodal interface useful, particularly the image updates, indicating a preference for visual feedback. However, they also expressed a desire for more visual indicators for system status (e.g., to know if the system is processing) and more frequent image updates.
- Expertise Influences Plan Quality: A correlation was found between the operator's expertise and the quality of the generated plan, suggesting that experienced users can produce better outcomes.
Key Results
- Corpus: A corpus of 22 multimodal dialogs was collected and is available for research.
- Dialog Act Analysis: The "inform" dialog act was the most frequent user action (54%), used for providing plan slot values. "Image" dialog acts (requesting or sending plan images) accounted for approximately 16% of interactions, highlighting their importance.
- Plan Quality: At least 45% of the plans generated were rated as "High Quality" or "Very High Quality" by an expert. Mission 1 (Category A, surveying) generally resulted in lower-rated plans and took longer than Mission 2 (Category B, reacquisition).
- User Satisfaction: The average user satisfaction score was 3.5 out of 5. Users found the images useful but suggested more frequent updates. They also noted that the WoZ setup introduced delays, which would not be present in a fully implemented system.
Limitations and Future Work
- Wizard-of-Oz Limitations: The authors acknowledge that the WoZ setup introduces response delays, which are not representative of a final, automated system.
- Data Collection Expansion: Future work includes collecting more data, potentially using Amazon Mechanical Turk with Open Street Maps to increase audience diversity and address gender imbalance.
- Developing the Conversational Agent: The collected corpus will be used to train a data-driven, mixed-initiative conversational agent for collaborative mission planning. This agent will leverage Reinforcement Learning, with reward functions derived from user preferences for plan quality and speed.
- Alternative Learning Methods: The authors also suggest exploring supervised approaches like Hybrid Code Networks (HCN) for systems that require less data.
- Further Evaluation: The developed system will be compared against a baseline in future human evaluation studies.
Summary updated