SiSCo: Signal Synthesis for Effective Human-Robot Communication Via Large Language Models

Shubham D. Sonawani, Fabian Weigend, Heni Ben Amor

TL;DR

This paper introduces SiSCo, a novel framework that leverages Large Language Models (LLMs) and mixed reality to automatically generate visual signals for human-robot collaboration. SiSCo significantly improves task efficiency and success rates compared to natural language, while also reducing cognitive load for human operators.

What Problem Does This Paper Address?

Effective human-robot collaboration (HRC) relies heavily on clear and intuitive communication. While visual signals are highly effective for conveying information quickly, designing and producing them is resource-intensive and requires specialized expertise. This often leads to static, pre-defined signals that lack adaptability to dynamic environments or novel situations. The paper addresses this gap by exploring how LLMs can be used to dynamically synthesize context-aware visual cues on-the-fly, bridging the divide between textual understanding and visual communication for HRC.

Methods

The core of SiSCo is a framework that integrates LLMs with mixed-reality technologies. The system takes a "task prompt" as input, which includes details about the task, objects, and desired outcomes. This prompt is processed through a hierarchical pipeline of LLM function calls:

  • Task Manager: Parses the input prompt and generates sub-prompts for other modules.
  • Natural Language Signal Synthesizer (NLSS): Summarizes the task into concise bullet points for natural language instructions.
  • Object Visual Signal Synthesizer (ObjVSS): Generates SVG code for visual representations (icons) of objects, based on their descriptions and colors.
  • Instruction Visual Signal Synthesizer (InstVSS): Generates SVG code for visual instructions, including object trajectories and orientations, based on goal positions and rotations.

The generated SVG code is then used to create visual signals, which can be displayed via a projector onto the workspace (VSIntPro) or on a monitor (VSM). The system utilizes GPT-4-Turbo as its LLM, chosen for its superior signal generation capabilities.

Datasets

The paper does not explicitly mention training datasets for the LLMs, as it leverages pre-trained LLMs (specifically GPT-4-Turbo). The evaluation was conducted using a custom human-robot teaming task involving the assembly of six distinct objects into various structures. Participants encountered six randomized assembly problems, with each problem presented using one of three SiSCo signal modalities (NLS, VSM, VSIntPro). A questionnaire study was also conducted, using real-world objects and their corresponding generated visual signals for evaluation.

Key Insights and Ideas

  • LLMs as Visual Signal Generators: The paper demonstrates that LLMs, primarily known for text generation, can be effectively repurposed to generate structured visual information like SVG code for robotic signals. This opens up new avenues for AI in HRI beyond natural language.
  • On-the-Fly Synthesis: SiSCo's ability to generate signals dynamically, rather than relying on pre-defined libraries, allows for greater adaptability to novel objects and situations, a significant improvement over traditional methods.
  • Mixed Reality Integration: The use of mixed reality (specifically projection-based) enhances the intuitiveness and spatial relevance of the visual signals, directly overlaying them onto the task environment.
  • SVG as an Efficient Representation: SVG is highlighted as an effective format for visual signals due to its scalability, human-interpretability, and efficient abstraction of complex visual information compared to raw RGB data.

Key Results

The study found significant improvements when using SiSCo's visual signals compared to natural language:

  • Task Efficiency: Task completion time was reduced by approximately 73%, and comprehension time was reduced by up to 56% with VSIntPro compared to NLS.
  • Task Success: Task success rates increased by 18% with VSIntPro and 3% with VSM compared to NLS.
  • Cognitive Load: Participants experienced a 46% reduction in cognitive load (measured by NASA-TLX subscales) when using VSIntPro compared to NLS.
  • User Preference: Participants ranked VSIntPro as the most effective communication mode, followed by VSM, and then NLS.
  • Generalization: SiSCo received above-average user ratings for on-the-fly signals generated for unseen objects, demonstrating its generalization capabilities. The System Usability Scale (SUS) score of 82.0 indicates robust usability.

Limitations and Future Work

  • LLM Dependency: The current implementation relies on OpenAI's server infrastructure, creating a dependency. Future work aims to fine-tune LLMs for local execution to mitigate this limitation.
  • LLM Research Advancement: The paper acknowledges that as LLM research progresses, the capabilities of SiSCo can be further enhanced.
  • Further Exploration of Modalities: While the paper focuses on visual signals, future work could explore richer multimodal communication strategies.
  • Real-world Deployment: While the study demonstrates effectiveness in a controlled lab setting, further validation in more complex and dynamic real-world scenarios would be beneficial.

Summary updated

Abstract

Effective human-robot collaboration hinges on robust communication channels, with visual signaling playing a pivotal role due to its intuitive appeal. Yet, the creation of visually intuitive cues often demands extensive resources and specialized knowledge. The emergence of Large Language Models (LLMs) offers promising avenues for enhancing human-robot interactions and revolutionizing the way we generate context-aware visual cues. To this end, we introduce SiSCo–a novel framework that combines the computational power of LLMs with mixed-reality technologies to streamline the creation of visual cues for human-robot collaboration. Our results show that SiSCo improves the efficiency of communication in human-robot teaming tasks, reducing task completion time by approximately 73% and increasing task success rates by 18% compared to baseline natural language signals. Additionally, SiSCo reduces cognitive load for participants by 46%, as measured by the NASA-TLX subscale, and receives above-average user ratings for on-the-fly signals generated for unseen objects. To encourage further development and broader community engagement, we provide full access to SiSCo’s implementation and related materials on our GitHub repository.1
BibTeX
@inproceedings{sonawani2024sisco,
  author = {Shubham D. Sonawani and Fabian Weigend and Heni Ben Amor},
  title = {{SiSCo: Signal Synthesis for Effective Human-Robot Communication Via Large Language Models}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=UPPkGq1vAU},
}
Figure