Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task

Hassan Ali, Philipp Allgeuer, Stefan Wermter

Abstract

Human intention-based systems enable robots to perceive and interpret user actions to interact with humans and adapt to their behavior proactively. Therefore, intention prediction is pivotal in creating a natural interaction with social robots in human-designed environments. In this paper, we examine using Large Language Models (LLMs) to infer human intention in a collaborative object categorization task with a physical robot. We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions in a hierarchical architecture. Our evaluation of five LLMs shows the potential for reasoning about verbal and non-verbal user cues, leveraging their context-understanding and real-world knowledge to support intention prediction while collaborating on a task with a social robot.

Introduction

For social robots to transition from simple tools to collaborative partners, they must move beyond executing explicit commands and begin predicting human intent. Human communication is inherently multimodal; we seamlessly blend verbal instructions with non-verbal cues like hand gestures and facial expressions to convey our goals.

Traditional intention-prediction systems often suffer from "brittleness," requiring massive, specialized datasets and remaining restricted to specific environments. While Large Language Models (LLMs) offer sophisticated reasoning capabilities, they are primarily text-based. The core challenge addressed in this work is creating a reliable bridge between raw sensory data—what a robot sees and hears—and the high-level abstract reasoning required to understand the "why" behind human actions.

Proposed Architecture

The researchers developed a hierarchical framework utilizing the NICOL robot, a semi-humanoid platform. This architecture is divided into distinct layers that interface with an LLM and the robot's middleware to facilitate real-time collaboration.

The hierarchical system architecture showing the flow from core sensory inputs to robot actions through LLM-mediated reasoning.

Perceptive Reasoning: The Vision-to-Text Bridge

The first layer, Perceptive Reasoning, is responsible for translating raw vision and audio data into natural language "tokens" that an LLM can process.

  • Vision: The system uses MediaPipe to detect 3D keypoints for hand gestures (pointing, holding), body poses, and facial expressions.
  • Objects: An open-vocabulary detector (ViLD) identifies items in the environment.
  • Speech: OpenAI’s Whisper handles Automatic Speech Recognition (ASR).

The system then converts these detections into descriptive sentences. For example, raw data like [right hand, point, apple] is transformed into a descriptive prompt: "The user is indicating the apple with their right hand."

Detailed view of the Perceptive Reasoning layer, illustrating how detection and recognition modules generate natural language descriptions of human cues for the LLM.

Task Reasoning

The second layer combines these descriptions with interaction history and specific "task prompts" (the rules of the interaction). The LLM reasons about the user's goal within the context of the environment and outputs both verbal responses and physical robot actions (e.g., ACTION: Give apple).

Experimental Evaluation

The system was evaluated using five LLMs: GPT-4, two versions of GPT-3.5, Vicuna-1.3-33b, and Mistral-7b-instruct. Instead of a static dataset, the study employed a live Object Categorization Task as a benchmark.

The setup involved six objects with varying properties (color, shape, and purpose, such as fruit vs. container). In 150 trials, the robot had to observe a user's initial moves and infer the underlying categorization logic—such as sorting by color or by function—without being explicitly told the rule.

Key Results and Insights

  • LLMs as Multimodal Reasoners: The study demonstrates that native multimodal LLMs are not strictly necessary for robotics if a robust "vision-to-text" perceptive layer is implemented. Standard text-based LLMs can effectively reason about physical social cues when they are grounded in descriptive language.
  • Emergent Rule Inference: GPT-4 showed remarkable flexibility, successfully switching between sorting by "usage" or "appearance" based on the user's context. It achieved near-perfect scores in categorization and explaining its reasoning.
  • Performance Disparity: While smaller models like Mistral and Vicuna were competent at interpreting individual gestures (Perceptive Reasoning), they struggled to apply that information to the broader task (Task Reasoning), highlighting that prompt adherence is a primary bottleneck for smaller models.
  • The Spatial Reasoning Bottleneck: A consistent failure across all models was spatial orientation. LLMs occasionally confused left and right in 3D space, indicating a need for better grounding in physical coordinates.

Conclusion

This research highlights the potential of LLMs to serve as the reasoning backbone for social robots. By bridging the gap between perception and high-level reasoning, robots can proactively assist humans in complex, collaborative tasks. Future work will focus on improving spatial grounding and reducing the latency introduced by LLM API calls to enable smoother real-time interactions.

Summary updated

Abstract

Human intention-based systems enable robots to perceive and interpret user actions to interact with humans and adapt to their behavior proactively. Therefore, intention prediction is pivotal in creating a natural interaction with social robots in human-designed environments. In this paper, we examine using Large Language Models (LLMs) to infer human intention in a collaborative object categorization task with a physical robot. We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions in a hierarchical architecture. Our evaluation of five LLMs shows the potential for reasoning about verbal and non-verbal user cues, leveraging their context-understanding and real-world knowledge to support intention prediction while collaborating on a task with a social robot. Video: https://youtu.be/tBJHfAuzohI
BibTeX
@inproceedings{ali2024comparing,
  author = {Hassan Ali and Philipp Allgeuer and Stefan Wermter},
  title = {{Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task}},
  booktitle = {ICSR + AI},
  year = {2024},
  doi = {10.1007/978-981-96-3525-2\_25},
}
Figure