Talk to Act: A Synthetic Multi-Turn Dialogue Benchmark for Human Intent Extraction and Action Planning in Robotics

Deepu Krishnareddy, Mohammad Haziq Khan, Michael Danner, Matthias RätscharXiv 2026 · arXiv

TL;DR

The "Talk to Act" paper introduces a new synthetic benchmark for multi-turn human-robot interaction (HRI), focusing on extracting structured intent from natural, ambiguous dialogue. By framing intent extraction as a tool-calling problem, the authors demonstrate that small, open-weight language models (ranging from 270M to 8B parameters) can effectively plan robotic tasks without any specialized fine-tuning.

What Problem Does This Paper Address?

Most current robots are "command-driven"—they expect rigid, specific instructions like "Move to (x,y), Pick Object_A." However, humans don't talk like that. We use multi-turn dialogue, we are often ambiguous ("pick up that one"), and we revise our goals mid-sentence.

While massive frontier models (like GPT-4 or Gemini) are great at handling this complexity, they are often impractical for robotics due to high latency, privacy concerns, and the need for constant internet connectivity. There is a critical need for small, locally deployable models that can handle the "interaction layer" of robotics. The bottleneck has been a lack of high-quality, multi-turn datasets specifically designed to train and evaluate these models in a robotic context.

Methods

The authors propose a framework where a dialogue system acts as a "bridge" between natural language and a robot's perception/execution stack.

Overview of the proposed HRI framework showing how multi-turn dialogue is parsed into structured JSON for robot execution.

The key technical approach is framing intent extraction as a tool-calling problem. Instead of just generating text, the language model is prompted to invoke specific functions (e.g., pick, place, pour, handoff) with structured JSON arguments. These arguments include:

  • Action Intent: The specific task to perform.
  • Object Intent: Detailed descriptions of the target and source objects, including properties like color, shape, and material.
  • Grounding Phrases: Natural language descriptions that can be fed directly into modern open-vocabulary perception models like Grounded DINO or SAM 3.

Datasets

The authors introduced the Talk to Act dataset, a synthetically generated corpus of 977 multi-turn conversations.

The synthetic data generation pipeline using Qwen 3.6 and an LLM-based verification step.
  • Generation: They used Qwen 3.6 to generate dialogues across 40 unique combinations of user personas, tones, and environments (homes, warehouses, hospitals).
  • Complexity: Each conversation includes at least ten turns, featuring chitchat, task requests, and—crucially—ambiguity resolution (e.g., the robot asking for clarification when a user says "the blue one" in a room with two blue mugs).
  • Verification: To ensure quality, the data underwent an automated "LLM-as-a-judge" verification pass to fix logical inconsistencies or state-tracking errors (like a robot trying to "pick" an object it is already holding).

Key Insights and Ideas

  • The Power of Small Models: One of the most surprising findings is that "tiny" models, like FunctionGemma-270M, can produce structured, useful outputs for robotics. While they aren't as accurate as 4B or 8B models, their ability to follow a JSON schema out-of-the-box is a strong signal for edge-computing applications.
  • Context is King: The benchmark highlights that intent extraction isn't just about the current sentence; it's about the history. Resolving "the other one" requires the model to track what was discussed three turns ago.
  • Bridging the Semantic Gap: By extracting "grounding descriptions" (e.g., "the large blue water jug on the side table"), the system provides exactly what modern vision models need to find objects in a scene, moving away from limited, closed-vocabulary object lists.

Key Results

The authors benchmarked five open-weight models: FunctionGemma (270M), Gemma4 (2B and 4B), and Ministral (3B and 8B).

Fig. 4. Comparison of five models (FunctionGemma-270M, Gemma4:E2B, Gemma4:E4B, Ministral3:3B, and Ministral3:8B) on Goal Accuracy, Action Accuracy, and Action F1. Gemma4:E4B scores highest on all three metrics (63.6%, 59.0%, and 74.2%). FunctionGemma-270M scores lowest on Goal Accuracy and Action Accuracy, and Ministral3:3B scores lowest on Action F1.
  • Top Performer: Gemma4:E4B achieved the best results, with a 63.6% Goal Achievement Accuracy and a 74.2% Action-level F1 score.
  • Zero-Shot Capability: All models were tested "out of the box" without task-specific fine-tuning. The fact that a 4B parameter model can successfully plan a multi-step task over 60% of the time suggests that fine-tuning on this specific dataset could push performance much higher.
  • Efficiency vs. Scale: Interestingly, the 4B Gemma model outperformed the 8B Ministral model in several metrics, suggesting that model architecture and pre-training data for function calling might be more important than raw parameter count for this specific task.

Limitations and Future Work

The authors acknowledge several areas for growth:

  1. Synthetic vs. Real: While synthetic data allows for scale, it may lack the "messiness" and unpredictable errors of real human speech.
  2. Action Breadth: The current benchmark is limited to four basic actions. Real-world robotics requires a much broader library of skills.
  3. Closed-Loop Feedback: Currently, the robot doesn't "talk back" about its physical state (e.g., "I can't reach that mug"). Future iterations aim to incorporate robot state feedback into the dialogue.
  4. Multimodality: The next step is integrating vision directly into the intent extraction pipeline, allowing the model to "see" the ambiguity it is trying to resolve.

Summary updated

Abstract

Multi-turn conversational human-robot interaction (HRI) continues to be a challenging problem in robotics, as many existing systems rely on rigid, command-driven interfaces that can be unintuitive for users. Recent advances in Large Language Models (LLMs) provide a promising direction, given their ability to process natural language, generate structured outputs, and support tool-based reasoning, all of which are relevant for interpreting user intent in robotic settings. However, progress in this area is constrained by the limited availability of datasets and benchmarks that capture the nuances of real-world human communication, such as multi-turn dialogue, ambiguity, and context-dependent requests. In this work, we present a synthetically generated multi-turn dialogue dataset for HRI, annotated with action intents, object references, and task plans across a range of scenarios and user personas. Our primary focus is on evaluating the capability of small, open-weight language models for robotic tasks, particularly in settings where local deployment and efficiency are important. To this end, we frame intent extraction as a tool-calling problem where each predicted action corresponds to a structured function call with arguments grounded in the dialogue and benchmark models ranging from 270M to 8B parameters on their ability to perform intent extraction and action planning without task-specific fine-tuning. Our results show that these models can achieve reasonable performance on the task even at relatively small scales and without any task-specific fine-tuning. This suggests that compact, locally deployable models already provide a useful baseline for conversational robotic systems. We expect that task-specific fine-tuning and more realistic training data will enhance performance further and make such models increasingly suitable for real-world robotic applications.
BibTeX
@misc{krishnareddy2026talk,
  author = {Deepu Krishnareddy and Mohammad Haziq Khan and Michael Danner and Matthias Rätsch},
  title = {{Talk to Act: A Synthetic Multi-Turn Dialogue Benchmark for Human Intent Extraction and Action Planning in Robotics}},
  year = {2026},
  eprint = {2611.17549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17549},
}
Figure