Comparing Apples to Oranges: LLM-Powered Multimodal Intention Prediction in an Object Categorization Task

Hassan Ali, Philipp Allgeuer, Stefan Wermter

TL;DR

This paper introduces a hierarchical framework that enables social robots to predict human intentions by feeding multimodal cues—such as gestures, poses, and speech—into Large Language Models (LLMs). By using the LLM as a reasoning backbone, the robot can autonomously infer complex task rules (like object categorization) and collaborate with humans in real-time without needing task-specific training data.

What Problem Does This Paper Address?

For social robots to be truly useful, they must move beyond following explicit commands and start predicting human intent. Human communication is inherently multimodal; we use a mix of verbal instructions, hand gestures, and facial expressions to convey goals.

Traditional intention-prediction systems are often "brittle"—they require massive, specialized datasets and are typically restricted to very specific environments or modalities. While LLMs offer incredible reasoning capabilities, they are primarily text-based. The challenge lies in creating a reliable bridge between raw sensory data (what the robot sees and hears) and the high-level abstract reasoning required to understand why a human is pointing at a banana while saying "I'm hungry."

Methods

The researchers utilized the NICOL robot (a semi-humanoid platform) and developed a two-layered reasoning architecture:

  1. Perceptive Reasoning: This layer translates raw vision and audio data into natural language.
  2. Vision: Uses MediaPipe to detect 3D keypoints for hand gestures (pointing, holding, resting), body poses (tracking user location), and facial expressions.
  3. Speech: Uses OpenAI’s Whisper for Automatic Speech Recognition (ASR).
  4. Objects: Uses ViLD, an open-vocabulary object detector, to identify items on the table.
  5. The Bridge: An LLM takes these "tokens" (e.g., [right hand, point, apple]) and converts them into a descriptive sentence: "The user is indicating the apple with their right hand."
  6. Task Reasoning: This layer combines the descriptive text from the first layer with the interaction history and "task prompts" (the rules of the interaction). The LLM then reasons about the user's goal and outputs robot actions (e.g., SAY: "I'll put the fruit here" or ACTION: Move apple).

The system was evaluated using five different LLMs: GPT-4, two versions of GPT-3.5, Vicuna-1.3-33b, and Mistral-7b-instruct.

Datasets

The study did not rely on a static, pre-existing dataset for training. Instead, it used a live, collaborative Object Categorization Task as a benchmark:

  • Setup: Six objects with varying properties (color: red/yellow; shape: round/cylindrical; purpose: fruit/container).
  • Evaluation: 150 trials covering every possible pair of objects as "category initiators."
  • Metrics: The researchers measured accuracy across three phases: Task Understanding (Phase 1), Object Categorization (Phase 2), and Intention Explanation (Phase 3).

Key Insights and Ideas

  • LLMs as Multimodal Translators: The most significant "aha" moment is that we don't necessarily need a native multimodal LLM to do robotics. By using a "vision-to-text" perceptive layer, standard text-based LLMs can effectively "see" and reason about physical social cues.
  • Emergent Rule Inference: The robot was not explicitly told the categorization rules (e.g., "Sort by color"). Instead, it had to observe the user's first few moves and infer the underlying logic. GPT-4 proved remarkably flexible, switching between sorting by "usage" (fruits vs. containers) or "appearance" (red vs. yellow) based on context.
  • The Grounding Gap: Smaller, open-weight models (Mistral and Vicuna) were surprisingly good at interpreting gestures (Perceptive Reasoning) but struggled to apply that information to the overall task (Task Reasoning). They could see the user pointing but couldn't consistently connect that point to the categorization goal.

Key Results

  • GPT-4 was the top performer: It achieved near-perfect scores in Phase 2 (Categorization) and Phase 3 (Explaining its reasoning), showing a deep understanding of human intent.
  • GPT-3.5 Bias: The GPT-3.5 models showed a slight bias toward "usage" (function) over physical attributes like color, occasionally leading to errors when the user intended to sort by appearance.
  • Spatial Struggles: A common failure across all models was spatial reasoning. LLMs occasionally "swapped" left and right in their internal logic, suggesting that while they understand what is happening, they still struggle with the where in a 3D coordinate system.
  • Correlation: There was a strong positive correlation between a model's performance in Phase 1 (understanding the prompt) and its success in the actual task, highlighting that "prompt adherence" is the primary bottleneck for smaller models.

Limitations and Future Work

  • Spatial Orientation: The authors acknowledge that LLM spatial reasoning is not yet performant enough for complex manipulation without better grounding.
  • Real-time Latency: While the vision pipelines are fast, the round-trip time for LLM API calls can introduce delays in social interaction.
  • Future Directions: The team plans to test the system's scalability across a wider range of interactive scenarios and explore whether native multimodal LLMs (like GPT-4o or Gemini) can eventually replace the manual "vision-to-text" bridge used here.

Summary updated

Abstract

Human intention-based systems enable robots to perceive and interpret user actions to interact with humans and adapt to their behavior proactively. Therefore, intention prediction is pivotal in creating a natural interaction with social robots in human-designed environments. In this paper, we examine using Large Language Models (LLMs) to infer human intention in a collaborative object categorization task with a physical robot. We propose a novel multimodal approach that integrates user non-verbal cues, like hand gestures, body poses, and facial expressions, with environment states and user verbal cues to predict user intentions in a hierarchical architecture. Our evaluation of five LLMs shows the potential for reasoning about verbal and non-verbal user cues, leveraging their context-understanding and real-world knowledge to support intention prediction while collaborating on a task with a social robot. Video: https://youtu.be/tBJHfAuzohI
BibTeX
@inproceedings{ali2024comparing,
  author = {Hassan Ali and Philipp Allgeuer and Stefan Wermter},
  title = {{Comparing Apples to Oranges: LLM-Powered Multimodal Intention Prediction in an Object Categorization Task}},
  booktitle = {ICSR + AI},
  year = {2024},
  doi = {10.1007/978-981-96-3525-2\_25},
}

Loading related content...

Figure