Abstract
Large language models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing tasks. However, a persistent challenge remains: their tendency to generate incorrect or fabricated responses when faced with queries that exceed their inherent knowledge or physical capabilities. This paper investigates the concept of "infeasibility"—tasks that an LLM cannot perform due to its nature as a text-based software entity. We propose a taxonomy of four infeasible task categories, develop a comprehensive benchmark for evaluation, and explore fine-tuning strategies to enhance a model's ability to recognize and refuse such requests.
Introduction: The Helpfulness Trap
Current LLM training paradigms, particularly Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), prioritize "helpfulness." While this makes models more useful for standard queries, it inadvertently conditions them to be over-cooperative. When a user asks an LLM to perform a physical task, such as "dust the bookshelf in the living room," the model often fails to recognize its own physical limitations. Instead of declining, it may hallucinate a confirmation, leading to a breach of trust and potential safety concerns in integrated systems.
This research addresses the need for LLMs to maintain "self-awareness" regarding their operational boundaries, ensuring they can distinguish between what they can and cannot do.
A Taxonomy of Infeasibility
To systematically study this problem, we categorize infeasible tasks into four primary domains:
- Physical Interaction: Tasks requiring a physical body or presence in the real world (e.g., "Change my car tire").
- Virtual Interaction: Tasks requiring external digital tools, real-time web access, or account-level permissions (e.g., "Book a flight for me").
- Non-text Input/Output: Requests involving the processing or generation of images, video, or audio for models that are strictly text-to-text.
- Self-awareness: Tasks that assume the model possesses consciousness, personal feelings, or a subjective worldview.
Methodology: Building a Self-Aware Model
The research follows a multi-stage pipeline to generate data and train models to recognize these limits.
Data Generation
Using GPT-4 as a seed generator, we curated a dataset of 1,898 infeasible tasks across the four categories, paired with 1,850 feasible control tasks. This benchmark was carefully balanced for length to ensure models weren't simply using prompt length as a proxy for feasibility.
Verbalized Confidence Elicitation
We first evaluated how well models could "self-diagnose" their capabilities through prompting. We tested four strategies: Pre-response (asking before answering), Mid-response (categorizing first), Post-response (reflecting after answering), and Mix-response. Surprisingly, the simplest "Pre-response" method proved most effective, outperforming more complex reasoning chains.
Selection-Based Fine-Tuning
The core of our training approach is "Selection-based" fine-tuning. Rather than just adding new refusal examples to a training set, we identify specific instances in existing datasets (like Alpaca) where the base model generates "dishonest" or hallucinated responses to infeasible queries. We then replace these specific outputs with structured refusals that include a decline, a reason, and a helpful suggestion.
Key Results and Insights
Our experiments with LLaMA-2-7b and 70b yielded several critical findings:
- Significant Improvement: After selection-based fine-tuning, the LLaMA-2-7b model's refusal rate on out-of-distribution (OOD) infeasible tasks increased from 13% to 73.5%, surpassing the zero-shot performance of GPT-4 (58.5%).
- The Helpfulness-Refusal Trade-off: There is a measurable "tax" on teaching refusal. Models trained to be more cautious showed a slight decline in performance on standard helpfulness benchmarks, occasionally refusing subjective prompts they were actually capable of answering.
- GPT-4 Calibration: GPT-4 exhibits high "sharpness" in its confidence scores, meaning it is generally very certain about whether a task is feasible or not, whereas smaller models tend to be overconfident in their hallucinations.
Conclusion
Recognizing the limits of one's own capabilities is a hallmark of intelligence. By moving beyond simple "helpfulness" and training LLMs to recognize infeasibility, we can create more reliable and trustworthy AI systems. Future work will focus on refining the taxonomy of infeasibility and finding the optimal balance between helpfulness and honest refusal.
Summary updated