Abstract: Xuan Zhang, Yongliang Shen, Zhe Zheng, Linjuan Wu, Wenqi Zhang, Yuchen Yan, Qiuying Peng, Jun Wang, Weiming Lu. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
2025.
Introduction
Most research into tool-augmented Large Language Models (LLMs) operates under an idealistic assumption: that user queries are always explicit and unambiguous. In real-world applications, however, user requests are often messy, providing incomplete information (e.g., "Book me a ride" without specifying a destination) or ambiguous intents. When faced with these "unspecified queries," current LLMs frequently hallucinate missing parameters or fail to complete the task.
Existing attempts to solve this via interactive clarification—where the model asks the user for more information—face two major hurdles: data scarcity and error accumulation. Creating high-quality datasets for clarification requires expensive human annotation. Furthermore, if a model makes a single mistake in a multi-turn dialogue, it often lacks the training to recover, leading to task failure. ASKTOACT is a novel framework designed to address these challenges by automating data generation and introducing a robust self-correction mechanism.
The ASKTOACT Framework
ASKTOACT tackles the limitations of current tool-use models through two primary innovations: automated reverse-engineered data generation and self-correction training.
1. Automated Data Generation
Instead of relying on human-written vague queries, the researchers developed a method to systematically "break" existing datasets of perfect queries. By identifying the parameters required for an API call (e.g., location and food_type) and stripping them from the original user query, they created a "vague" query with a known "ground truth" of what information is missing. This approach allowed for the automated generation of over 35,000 high-quality training examples.
2. Self-Correction Training
To prevent models from failing after an initial error, ASKTOACT intentionally injects errors into training dialogues. The framework identifies five common error types, such as Redundant Clarification (asking for information already provided) and Irrelevant Clarification. The model is trained using special tokens (<SOE> and <EOE>) to demarcate these errors. Through selective masking during fine-tuning, the model learns to recognize its mistakes, apologize, and provide the correct response without learning to repeat the erroneous behavior.
Experimental Setup
The study utilized two primary datasets modified for Intent Clarification (IC):
- xlam-IC: Derived from the
xlam-function-calling-60kdataset, containing 29,821 training samples across 21 functional categories. - Taskbench-IC: Used for Out-of-Domain (OOD) testing to evaluate the model's ability to generalize to entirely unseen APIs.
Results and Analysis
The experimental results demonstrate that ASKTOACT significantly enhances the robustness and efficiency of LLMs in tool-use scenarios. A fine-tuned Qwen2.5-7B model using the ASKTOACT strategy achieved performance comparable to or better than much larger models like GPT-4o.
Key Findings:
- Intent Recovery: ASKTOACT recovered over 57% of critical unspecified intents, a significant improvement over baseline models.
- Efficiency: Clarification efficiency improved by an average of 10.46%, requiring fewer turns to reach the correct tool call.
- Accuracy: The framework achieved over 81% tool selection accuracy and 68% parameter resolution accuracy.
- Generalization: In OOD testing, the model successfully handled unseen APIs, proving it learned the underlying logic of clarification rather than simple memorization.
- Human Satisfaction: In real-world testing, ASKTOACT variants achieved a user satisfaction score of 4.61/5, compared to 3.80/5 for the base model.
Conclusion and Future Work
ASKTOACT demonstrates that specialized data strategies and self-correction mechanisms can enable smaller models to outperform larger counterparts in complex tool-use tasks. By treating tool parameters as anchors for user intent, the framework provides a mathematically grounded way for models to interact with users.
Future work will focus on expanding the diversity of error patterns beyond the five pre-defined types and exploring context-dependent queries where intent must be inferred from the results of previous API calls. Additionally, researchers aim to better account for human factors like user patience in long-form dialogues.
Summary updated