Abstract
Large language models (LLMs) have shown remarkable performance in various tasks but often fail to handle queries that exceed their knowledge and capabilities, leading to incorrect or fabricated responses. This paper addresses the need for LLMs to recognize and refuse infeasible tasks due to the requests surpassing their capabilities. We conceptualize four main categories of infeasible tasks for LLMs, which cover a broad spectrum of hallucination-related challenges identified in prior literature. We develop and benchmark a new dataset comprising diverse infeasible and feasible tasks to evaluate multiple LLMs' abilities to decline infeasible tasks. Furthermore, we explore the potential of increasing LLMs' refusal capabilities with fine-tuning. Our experiments validate the effectiveness of the trained models, suggesting promising directions for improving the performance of LLMs in real-world applications.
Introduction
Current Large Language Models (LLMs) are often trained to be maximally helpful, a goal achieved through Supervised Fine-Tuning (SFT). However, this drive for helpfulness often leads to "functional hallucination," where models confidently attempt to fulfill requests that are physically or logically impossible for a text-based AI. For instance, an LLM might provide detailed instructions on how it will "dust a bookshelf" despite lacking a physical presence.
This research addresses the critical need for LLMs to recognize their operational boundaries. By identifying functional infeasibility—tasks requiring physical bodies, real-time external tool access, or non-textual processing—we can move toward more reliable and honest AI agents.
Methodology and Taxonomy
The study establishes a systematic approach to identifying and addressing infeasibility through a three-pronged methodology: conceptual taxonomy, confidence elicitation, and refusal fine-tuning.
Conceptual Taxonomy
The researchers defined four primary categories of infeasible tasks:
- Physical Interaction: Tasks requiring a physical body (e.g., "Change my car tire").
- Virtual Interaction: Tasks requiring external digital tools or real-time web access (e.g., "Check local store inventory").
- Non-text I/O: Processing or creating audio, video, or sensory data.
- Self-awareness: Requests involving sentient-like reflection or internal states.
Dataset Generation Pipeline
To evaluate and train models, a new benchmark was developed using a rigorous pipeline. This involved literature review, task definition, and automated generation using GPT-4, followed by human verification and data balancing.
Training Strategies
Two fine-tuning strategies were explored to instill refusal behavior:
- Selection-based: Identifying and replacing infeasible tasks in existing instruction-tuning sets (like Alpaca) with refusal responses.
- Augment-based: Supplementing training sets with entirely new synthetic infeasible tasks.
Key Insights and Results
The experiments revealed a significant "Helpfulness Trap": standard SFT conditions models to ignore their limitations. While GPT-4 demonstrated a latent ability to distinguish feasible from infeasible tasks (achieving an AUROC of 0.967), it typically only exercises this judgment when explicitly prompted to do so.
Fine-tuning Performance
Fine-tuning smaller models like LLaMA-2-7B proved highly effective. Using the selection-based method, the refusal rate on out-of-distribution (OOD) benchmarks reached 73.5%, outperforming the baseline refusal rates of even GPT-4o and GPT-3.5. However, this improvement comes with a "Refusal Trade-off": models often become over-cautious, occasionally refusing valid tasks that mention external tools or platforms.
Conclusion and Future Directions
While this work significantly advances the ability of LLMs to recognize their limits, challenges remain. The current taxonomy is coarse, and models exhibit an "I Can't" bias that can hinder general helpfulness. Future research should focus on finer-grained taxonomies and extending these benchmarks to multimodal models (VLMs/ALMs), where the boundaries of feasibility are constantly shifting. Ensuring models are both honest about their limitations and helpful within their scope remains a primary goal for the development of reliable AI agents.
Summary updated