Recognizing Limits: Investigating Infeasibility in Large Language Models

Wenbo Zhang, Zihang Xu, Hengrui Cai

Abstract

Large language models (LLMs) have demonstrated exceptional capabilities across a wide range of natural language processing tasks. However, a persistent challenge remains: their tendency to generate incorrect or fabricated responses when faced with queries that exceed their inherent knowledge or physical capabilities. This paper investigates the concept of "infeasibility"—tasks that an LLM cannot perform due to its nature as a text-based software entity. We propose a taxonomy of four infeasible task categories, develop a comprehensive benchmark for evaluation, and explore fine-tuning strategies to enhance a model's ability to recognize and refuse such requests.

Introduction: The Helpfulness Trap

Current LLM training paradigms, particularly Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF), prioritize "helpfulness." While this makes models more useful for standard queries, it inadvertently conditions them to be over-cooperative. When a user asks an LLM to perform a physical task, such as "dust the bookshelf in the living room," the model often fails to recognize its own physical limitations. Instead of declining, it may hallucinate a confirmation, leading to a breach of trust and potential safety concerns in integrated systems.

Comparison of undesired (hallucinated) and desired (refusal with reason) LLM responses to an infeasible physical task.

This research addresses the need for LLMs to maintain "self-awareness" regarding their operational boundaries, ensuring they can distinguish between what they can and cannot do.

A Taxonomy of Infeasibility

To systematically study this problem, we categorize infeasible tasks into four primary domains:

  1. Physical Interaction: Tasks requiring a physical body or presence in the real world (e.g., "Change my car tire").
  2. Virtual Interaction: Tasks requiring external digital tools, real-time web access, or account-level permissions (e.g., "Book a flight for me").
  3. Non-text Input/Output: Requests involving the processing or generation of images, video, or audio for models that are strictly text-to-text.
  4. Self-awareness: Tasks that assume the model possesses consciousness, personal feelings, or a subjective worldview.

Methodology: Building a Self-Aware Model

The research follows a multi-stage pipeline to generate data and train models to recognize these limits.

The data generation and post-processing pipeline used to create the Infeasible Task Benchmark.

Data Generation

Using GPT-4 as a seed generator, we curated a dataset of 1,898 infeasible tasks across the four categories, paired with 1,850 feasible control tasks. This benchmark was carefully balanced for length to ensure models weren't simply using prompt length as a proxy for feasibility.

Verbalized Confidence Elicitation

We first evaluated how well models could "self-diagnose" their capabilities through prompting. We tested four strategies: Pre-response (asking before answering), Mid-response (categorizing first), Post-response (reflecting after answering), and Mix-response. Surprisingly, the simplest "Pre-response" method proved most effective, outperforming more complex reasoning chains.

Selection-Based Fine-Tuning

The core of our training approach is "Selection-based" fine-tuning. Rather than just adding new refusal examples to a training set, we identify specific instances in existing datasets (like Alpaca) where the base model generates "dishonest" or hallucinated responses to infeasible queries. We then replace these specific outputs with structured refusals that include a decline, a reason, and a helpful suggestion.

Key Results and Insights

Our experiments with LLaMA-2-7b and 70b yielded several critical findings:

  • Significant Improvement: After selection-based fine-tuning, the LLaMA-2-7b model's refusal rate on out-of-distribution (OOD) infeasible tasks increased from 13% to 73.5%, surpassing the zero-shot performance of GPT-4 (58.5%).
  • The Helpfulness-Refusal Trade-off: There is a measurable "tax" on teaching refusal. Models trained to be more cautious showed a slight decline in performance on standard helpfulness benchmarks, occasionally refusing subjective prompts they were actually capable of answering.
  • GPT-4 Calibration: GPT-4 exhibits high "sharpness" in its confidence scores, meaning it is generally very certain about whether a task is feasible or not, whereas smaller models tend to be overconfident in their hallucinations.

Conclusion

Recognizing the limits of one's own capabilities is a hallmark of intelligence. By moving beyond simple "helpfulness" and training LLMs to recognize infeasibility, we can create more reliable and trustworthy AI systems. Future work will focus on refining the taxonomy of infeasibility and finding the optimal balance between helpfulness and honest refusal.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable performance in various tasks but often fail to handle queries that exceed their knowledge and capabilities, leading to incorrect or fabricated responses.This paper addresses the need for LLMs to recognize and refuse infeasible tasks due to the requests surpassing their capabilities.We conceptualize four main categories of infeasible tasks for LLMs, which cover a broad spectrum of hallucination-related challenges identified in prior literature.We develop and benchmark a new dataset comprising diverse infeasible and feasible tasks to evaluate multiple LLMs' abilities to decline infeasible tasks .Furthermore, we explore the potential of increasing LLMs' refusal capabilities with fine-tuning.Our experiments validate the effectiveness of the trained models, suggesting promising directions for improving the performance of LLMs in real-world applications.
BibTeX
@misc{zhang2024recognizing,
  author = {Wenbo Zhang and Zihang Xu and Hengrui Cai},
  title = {{Recognizing Limits: Investigating Infeasibility in Large Language Models}},
  year = {2024},
  eprint = {2408.05873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05873},
}
Figure