How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey

Zhonghao Shi, Ellen Landrum, Amy O'Connell, Mina J. Kian, Leticia Pinto-Alva, Kaleen Shrestha, Xiaoyuan Zhu, Maja J. MataricAAAI 2024 · DOI

Abstract

Socially Assistive Robots (SARs) aim to provide cognitive and affective support to vulnerable populations, yet they often suffer from "social brittleness" due to their reliance on rigid, rule-based systems. This survey explores how Large Language Models (LLMs) can bridge the gap toward human-level social intelligence. By categorizing recent advancements into dialogue, multimodal understanding, and robot policies, the paper highlights how LLMs enable personalized interactions while addressing the critical safety and ethical risks inherent in their deployment.

Introduction

Socially Assistive Robots (SARs) are designed to assist individuals with special needs, including children with autism spectrum disorder (ASD), older adults with dementia, and individuals managing mental health challenges. Despite their potential, current SAR implementations are frequently limited by "social brittleness." Most systems rely on pre-defined scripts or "Wizard-of-Oz" setups, where a human operator controls the robot's responses behind the scenes.

The fundamental challenge lies in the fluid, multimodal, and highly contextual nature of human social interaction. Traditional machine learning models often fail to generalize to the diverse, out-of-distribution scenarios encountered in real-world therapy. This survey examines how LLMs might serve as a "social brain," providing the reasoning capabilities necessary for robots to understand and respond to human nuances effectively.

Technical Pillars of LLM-Enhanced SAR

The integration of LLMs into SAR can be organized into three primary technical pillars:

1. Natural Language Dialogue

LLMs facilitate a shift from rigid scripts to context-aware, spontaneous conversation. Unlike traditional systems, LLMs can maintain "long-horizon" context, allowing a robot to reference information shared earlier in a session. This capability is vital for building the therapeutic rapport necessary for effective assistance.

2. Multimodal User Understanding

By leveraging Multimodal Language Models (MLMs) such as GPT-4V or CLIP, robots can interpret a user’s emotional state through a combination of speech, facial expressions, and body language. A significant advantage of MLMs is their capacity for "zero-shot" or "few-shot" social reasoning, enabling robots to interpret social cues they were not specifically trained for by utilizing the broad common-sense knowledge embedded in the model.

3. LLMs as Robot Policies

Beyond verbal communication, LLMs are increasingly used to determine a robot’s physical actions, such as gestures and movements. This approach moves beyond discrete, pre-programmed action sets toward more continuous and human-like action spaces. This allows for better alignment between the robot's physical presence and the sentiment of its speech, making interactions feel more empathetic and less robotic.

The Data and Alignment Challenge

A critical finding of this survey is the inadequacy of current datasets. Most multimodal social understanding research relies on small, specialized datasets that follow the Independent and Identically Distributed (IID) assumption. However, in SAR contexts, every user’s needs and environments are unique, rendering the IID assumption ineffective.

Furthermore, the "alignment problem" is particularly acute in therapy. An LLM-powered robot must quickly align its personality, values, and feedback style with the specific individual it is assisting. Personalized SAR policies are essential for ensuring that the robot’s interventions are both appropriate and effective for the user's specific condition.

Early Successes in Application

Early research into LLM-powered SAR has shown promise in several areas:

  • Motivational Coaching: LLMs have been used to develop robotic coaches that successfully balance the delivery of informative content with emotional support.
  • Gesture Matching: Studies indicate that LLM-driven policies can synchronize a robot’s physical gestures with the emotional tone of its dialogue more effectively than rule-based systems.
  • Knowledge Integration: LLMs allow robots to function as instructional agents, combining vast general knowledge with the engaging, physical presence of a robot to facilitate learning.

Limitations and Ethical Considerations

Despite the transformative potential of LLMs, several "red flags" hinder autonomous deployment in clinical settings:

  • Safety and Hallucinations: In sensitive contexts like mental health, a "hallucinated" or incorrect response from an LLM could be confusing or harmful. Currently, there are no theoretical safety guarantees for LLM outputs.
  • Explainability: Clinical adoption requires transparency. We currently lack the tools to fully explain why an LLM chooses a specific social response, which is a significant barrier for healthcare providers.
  • Latency: Social interaction is highly sensitive to timing. The processing delays associated with large models can lead to user frustration and a breakdown of the social bond.

Conclusion and Future Directions

The survey concludes that while LLMs provide a powerful framework for social intelligence, future research must move toward navigating "spontaneous tasks"—such as helping a child manage an unexpected social conflict—rather than focusing solely on pre-defined therapeutic games. Addressing the trade-offs between model complexity, response latency, and safety remains the primary hurdle for the next generation of socially assistive robots.

Summary updated

BibTeX
@inproceedings{shi2024how,
  author = {Zhonghao Shi and Ellen Landrum and Amy O'Connell and Mina J. Kian and Leticia Pinto-Alva and Kaleen Shrestha and Xiaoyuan Zhu and Maja J. Mataric},
  title = {{How Can Large Language Models Enable Better Socially Assistive Human-Robot Interaction: A Brief Survey}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v3i1.31245},
}

Loading related content...

Figure