TL;DR
The "Talk to Act" paper introduces a new synthetic benchmark for multi-turn human-robot interaction (HRI), focusing on extracting structured intent from natural, ambiguous dialogue. By framing intent extraction as a tool-calling problem, the authors demonstrate that small, open-weight language models (ranging from 270M to 8B parameters) can effectively plan robotic tasks without any specialized fine-tuning.
What Problem Does This Paper Address?
Most current robots are "command-driven"—they expect rigid, specific instructions like "Move to (x,y), Pick Object_A." However, humans don't talk like that. We use multi-turn dialogue, we are often ambiguous ("pick up that one"), and we revise our goals mid-sentence.
While massive frontier models (like GPT-4 or Gemini) are great at handling this complexity, they are often impractical for robotics due to high latency, privacy concerns, and the need for constant internet connectivity. There is a critical need for small, locally deployable models that can handle the "interaction layer" of robotics. The bottleneck has been a lack of high-quality, multi-turn datasets specifically designed to train and evaluate these models in a robotic context.
Methods
The authors propose a framework where a dialogue system acts as a "bridge" between natural language and a robot's perception/execution stack.
The key technical approach is framing intent extraction as a tool-calling problem. Instead of just generating text, the language model is prompted to invoke specific functions (e.g., pick, place, pour, handoff) with structured JSON arguments. These arguments include:
- Action Intent: The specific task to perform.
- Object Intent: Detailed descriptions of the target and source objects, including properties like color, shape, and material.
- Grounding Phrases: Natural language descriptions that can be fed directly into modern open-vocabulary perception models like Grounded DINO or SAM 3.
Datasets
The authors introduced the Talk to Act dataset, a synthetically generated corpus of 977 multi-turn conversations.
- Generation: They used Qwen 3.6 to generate dialogues across 40 unique combinations of user personas, tones, and environments (homes, warehouses, hospitals).
- Complexity: Each conversation includes at least ten turns, featuring chitchat, task requests, and—crucially—ambiguity resolution (e.g., the robot asking for clarification when a user says "the blue one" in a room with two blue mugs).
- Verification: To ensure quality, the data underwent an automated "LLM-as-a-judge" verification pass to fix logical inconsistencies or state-tracking errors (like a robot trying to "pick" an object it is already holding).
Key Insights and Ideas
- The Power of Small Models: One of the most surprising findings is that "tiny" models, like FunctionGemma-270M, can produce structured, useful outputs for robotics. While they aren't as accurate as 4B or 8B models, their ability to follow a JSON schema out-of-the-box is a strong signal for edge-computing applications.
- Context is King: The benchmark highlights that intent extraction isn't just about the current sentence; it's about the history. Resolving "the other one" requires the model to track what was discussed three turns ago.
- Bridging the Semantic Gap: By extracting "grounding descriptions" (e.g., "the large blue water jug on the side table"), the system provides exactly what modern vision models need to find objects in a scene, moving away from limited, closed-vocabulary object lists.
Key Results
The authors benchmarked five open-weight models: FunctionGemma (270M), Gemma4 (2B and 4B), and Ministral (3B and 8B).
- Top Performer: Gemma4:E4B achieved the best results, with a 63.6% Goal Achievement Accuracy and a 74.2% Action-level F1 score.
- Zero-Shot Capability: All models were tested "out of the box" without task-specific fine-tuning. The fact that a 4B parameter model can successfully plan a multi-step task over 60% of the time suggests that fine-tuning on this specific dataset could push performance much higher.
- Efficiency vs. Scale: Interestingly, the 4B Gemma model outperformed the 8B Ministral model in several metrics, suggesting that model architecture and pre-training data for function calling might be more important than raw parameter count for this specific task.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Synthetic vs. Real: While synthetic data allows for scale, it may lack the "messiness" and unpredictable errors of real human speech.
- Action Breadth: The current benchmark is limited to four basic actions. Real-world robotics requires a much broader library of skills.
- Closed-Loop Feedback: Currently, the robot doesn't "talk back" about its physical state (e.g., "I can't reach that mug"). Future iterations aim to incorporate robot state feedback into the dialogue.
- Multimodality: The next step is integrating vision directly into the intent extraction pipeline, allowing the model to "see" the ambiguity it is trying to resolve.
Summary updated