Enhancing Situation Awareness through Model-Based Explanation Generation

Konstantinos Gavriilidis, Ioannis Konstas, Helen Hastie, Andrea Munafò, Wei PangPractical D2T - 2nd Workshop on Practical LLM-Assisted Data-to-Text Generation, Proceedings of the Workshop 2024 · OpenReview

Abstract: Robots are often deployed in remote locations for tasks such as exploration, where users cannot directly perceive the agent and its environment. For Human-In-The-Loop applications, operators must have a comprehensive understanding of the robot’s current state and its environment to take necessary actions and effectively assist the agent. In this work, we compare different explanation styles to determine the most effective way to convey real-time updates to users. Additionally, we formulate these explanation styles as separate fine-tuning tasks and assess the effectiveness of large language models in delivering in-mission updates to maintain situation awareness.

Introduction

When robots operate in remote or hazardous environments—such as the deep ocean or distant planetary surfaces—human operators cannot observe them directly. To assist the robot effectively in a Human-In-The-Loop (HITL) capacity, the operator requires high levels of Situation Awareness (SA). This involves perceiving the robot's current state, comprehending the rationale behind its decisions, and projecting its future actions.

Raw robot logs and telemetry are often too dense and technical for rapid human consumption during time-critical missions. While natural language explanations can bridge this gap, the optimal style of explanation for maintaining SA remains an open question. Furthermore, the robustness of current Large Language Models (LLMs) in generating these specialized explanations in domains like maritime robotics must be evaluated.

The XARLM Framework

To address these challenges, we developed the eXplainable Autonomous Robot Language Model (XARLM) framework. This framework is designed to transform complex vehicle states and user queries into human-readable explanations.

The XARLM framework processes vehicle states and user queries to generate causal, counterfactual, and contrastive explanations.

Our methodology involved simulating maritime mission scenarios using the MOOS-IvP simulator, involving Unmanned Surface Vehicles (USVs) and Underwater Vehicles (AUVs). We focused on three distinct explanation styles:

  • Causal: Explaining the current action based on mission logic (e.g., "Alpha is returning because it finished its task.").
  • Counterfactual: Answering "What if?" queries to explore alternative states (e.g., "If the return state were deactivated, the vessel would resume surveying.").
  • Contrastive: Answering "Why not?" queries to clarify why a specific action is not being taken (e.g., "Alpha is not surveying because its return state is active.").

Model Fine-Tuning and Evaluation

We fine-tuned three open-source LLMs—Llama2-7B, Mistral-7B, and Falcon-7B—using QLoRA, treating each explanation style as a separate downstream task. Performance was evaluated using standard translation metrics (BLEU, ROUGE, METEOR) alongside custom metrics for Semantic Accuracy (SA) and Semantic Precision (SP) to ensure the models accurately reflected robot states without hallucination.

Performance comparison of fine-tuned LLMs (Llama2-7B, Mistral-7B, Falcon-7B) across various metrics, showing Mistral-7B's superior accuracy and precision.

Among the 7B-parameter models tested, Mistral-7B consistently outperformed Llama2 and Falcon. After fine-tuning, Mistral-7B achieved a Semantic Accuracy of 0.975 and a Semantic Precision of 0.887 across all explanation types.

User Study Insights

Two studies involving 21 participants (primarily robotics experts) revealed that there is no "one size fits all" explanation style.

  • Causal explanations were found to be the "gold standard" for understanding internal logic, leading to the highest percentage of correct mission-related answers (76.19%).
  • Contrastive and counterfactual explanations were significantly better for spatial awareness. Contrastive explanations achieved a 77.77% accuracy rate for questions regarding the robot's environment, nearly double that of causal explanations.
  • The Verbosity Trap: Fine-tuning proved essential to prevent LLMs from becoming overly wordy, which otherwise increases the cognitive load on the operator.

Conclusion and Future Work

While LLMs demonstrate high accuracy, a "Preference Gap" remains: 70% of professional users still preferred domain-expert template-based explanations due to their extreme brevity and rigid clarity. Future research will focus on integrating multi-modal data (such as live maps), dynamically tailoring explanation styles based on query types, and incorporating more human-expert-labeled data to refine the professional tone of generated text.

Summary updated

Abstract

Robots are often deployed in remote locations for tasks such as exploration, where users cannot directly perceive the agent and its environment. For Human-In-The-Loop applications, operators must have a comprehensive understanding of the robot’s current state and its environment to take necessary actions and effectively assist the agent. In this work, we compare different explanation styles to determine the most effective way to convey real-time updates to users. Additionally, we formulate these explanation styles as separate fine-tuning tasks and assess the effectiveness of large language models in delivering in-mission updates to maintain situation awareness. The code and dataset for this work are available at: https://github.com/konsgavriil/explainable_robotics_lm.
BibTeX
@inproceedings{gavriilidis2024enhancing,
  author = {Konstantinos Gavriilidis and Ioannis Konstas and Helen Hastie and Andrea Munafò and Wei Pang},
  title = {{Enhancing Situation Awareness through Model-Based Explanation Generation}},
  booktitle = {Practical D2T - 2nd Workshop on Practical LLM-Assisted Data-to-Text Generation, Proceedings of the Workshop},
  year = {2024},
  url = {https://openreview.net/forum?id=swQcpaGpbj},
}
Figure