Defining Boundaries: A Spectrum of Task Feasibility for Large Language Models

Wenbo Zhang, Zihang Xu, Hengrui Cai

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in text generation and reasoning. However, a persistent challenge remains: their tendency to "hallucinate" feasibility. While hallucinations are often discussed in the context of factual accuracy, a broader issue exists in task infeasibility. Many user requests fall outside the inherent operational scope of a text-to-text model, such as requests for physical labor or real-time external interactions.

Desired vs. Undesired LLM responses to infeasible physical tasks.

When asked to perform a task like "dust the bookshelf," a model may confidently reply with a fabricated description of the process rather than acknowledging its lack of physical embodiment. To address this, researchers from UC Irvine have developed a framework and benchmark to evaluate and improve how LLMs recognize their own operational limits.

Taxonomy of Task Infeasibility

The researchers categorized task infeasibility into four primary domains:

  1. Physical Interaction: Tasks requiring a physical body (e.g., cleaning, moving objects).
  2. Virtual Interaction: Tasks requiring external tools or real-time web access beyond the model's training cutoff.
  3. Non-text Input/Output: Tasks involving the processing or generation of images, audio, or video in a strictly text-based environment.
  4. Self-awareness: Tasks that require the model to have internal states or genuine consciousness.

Methodology

The study followed a three-phase approach to benchmark and mitigate these failures:

The systematic pipeline for defining and generating the Infeasible Benchmark.

1. Taxonomy & Generation

Using the defined categories, the team utilized GPT-4 to generate a balanced "Infeasible Benchmark" consisting of 1,898 infeasible tasks and 1,850 feasible tasks. This dataset serves as a rigorous testing ground for a model's ability to decline out-of-distribution requests.

2. Verbalized Confidence Elicitation

To determine if models "know" their limits, the researchers tested four prompting strategies (Pre-response, Mid-response, Post-response, and Mix-response). These prompts required the model to provide a probability score (0.0 to 1.0) regarding its ability to fulfill an instruction. Interestingly, the "Pre-response" strategy—asking for confidence before attempting the task—performed as well as more complex reasoning chains, suggesting that advanced models often recognize infeasibility immediately.

3. Refusal-Augmented Fine-Tuning

The team explored three strategies to train models to refuse tasks autonomously:

  • Selection-based: Identifying uncertain instances in existing datasets and replacing answers with refusals.
  • Augment-based: Adding new infeasible tasks to the training set.
  • Random-based: A baseline replacing random samples with refusals.

Key Results and Insights

Model Performance

GPT-4 emerged as the gold standard, achieving the highest AUROC (0.967) and the lowest Brier Score (0.056), indicating superior calibration in distinguishing feasible from infeasible tasks. In contrast, models like LLaMA2-70b and PaLM2 struggled significantly more without explicit training.

The Impact of Fine-Tuning

Fine-tuning proved highly effective. Using the Selection-based method, the researchers improved LLaMA2-7b’s refusal rate on out-of-distribution tasks from a baseline of 13% to 73.5%. However, a "Goldilocks zone" exists; over-training for refusal can lead to a decline in overall helpfulness on standard benchmarks like Alpaca.

Category Difficulty

"Physical Interaction" tasks were the easiest for models to identify as infeasible. Conversely, "Virtual Interaction" tasks—those requiring real-time tools—were the most difficult. Models frequently confused their internal training data knowledge with the ability to perform live external actions.

Conclusion

For LLMs to become reliable assistants, they must move beyond simple question-answering and learn to recognize when a request exceeds their capabilities. While top-tier models like GPT-4 show promise, explicit refusal training is essential to prevent hallucinations in real-world applications. As models evolve into native multi-modal systems, the boundaries of feasibility will continue to shift, requiring dynamic benchmarks that evolve alongside model capabilities.

Summary updated

BibTeX
@misc{zhang2024defining,
  author = {Wenbo Zhang and Zihang Xu and Hengrui Cai},
  title = {{Defining Boundaries: A Spectrum of Task Feasibility for Large Language Models}},
  year = {2024},
  eprint = {2408.05873},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.05873},
}
Figure