InferAct: Inferring Safe Actions for LLM-Based Agents Through Preemptive Evaluation and Human Feedback

Haishuo Fang, Xiaodan Zhu, Iryna Gurevych

Abstract

A crucial requirement for deploying LLM-based agents in real-life applications is robustness against risky or irreversible mistakes. However, existing research lacks a focus on the preemptive evaluation of reasoning trajectories performed by LLM agents, leading to a gap in ensuring safe and reliable operations. To explore better solutions, this paper introduces InferAct, a novel approach that leverages the Theory-of-Mind capability of LLMs to proactively detect potential errors before critical actions are executed (e.g., "buy-now" in automatic online trading or web shopping). InferAct is also capable of integrating human feedback to prevent irreversible risks and enhance the actor agent's decision-making process. Experiments on three widely used tasks demonstrate the effectiveness of InferAct. The proposed solution presents a novel approach and concrete contributions toward developing LLM agents that can be safely deployed in different environments involving critical decision-making.

Introduction

As Large Language Model (LLM) agents evolve from simple conversational interfaces to autonomous entities capable of executing web transactions, managing household appliances, or running system code, the consequences of "misalignment" become increasingly severe. A misunderstanding of user intent could lead to irreversible actions, such as purchasing an expensive incorrect item or deleting critical data.

Current safety mechanisms generally fall into two categories: manual validation, which is labor-intensive and limits autonomy, or post-execution feedback, which is ineffective for high-stakes, irreversible actions. InferAct addresses this gap by providing a framework for preemptive evaluation, detecting and correcting misaligned actions before they are executed.

Methodology: The InferAct Framework

The core innovation of InferAct is the application of Theory-of-Mind (ToM)—the cognitive ability to attribute mental states and intentions to others. Rather than asking an agent to self-reflect (which often results in biased justifications), InferAct functions as an external observer that deduces the agent's intent from its behavior.

The framework consists of two primary components:

  1. Task Inference Unit: This unit analyzes the agent's sequence of actions and observations to infer what the agent believes its current task is.
  2. Task Verification Unit: This unit evaluates the inferred task against the user's original goal. It checks for completion alignment (does the inferred task fulfill the user's goal?) and progress evaluation (is the agent moving in the right direction?).
The workflow of InferAct involves an Actor Agent, the Inference/Verification units, and a human-in-the-loop for feedback.

InferAct is designed for efficiency, triggering only when the agent attempts a pre-defined critical action (e.g., "Buy Now" or "Clean").

Experimental Setup

The researchers evaluated InferAct across several interactive environments:

  • WebShop: An online shopping benchmark where "Buy Now" is the critical action.
  • ALFWorld: A household simulator focusing on actions like "Clean," "Heat," or "Cool."
  • HotPotQA: A multi-hop question-answering task where the final answer is the critical action.
  • R-judge: A safety benchmark for coding environments involving high-risk bash commands and API calls.

Key Insights and Results

InferAct demonstrates significant improvements over standard baselines, such as direct prompting or uncertainty-based detection methods.

Performance Metrics

  • Detection Accuracy: InferAct achieved up to a 20% improvement in Macro-F1 score for identifying misaligned actions.
  • Human-AI Synergy: In collaborative workflows, the framework reduced human oversight requirements by 50% with only a marginal 3% drop in task performance compared to full manual inspection.
  • Task Success: Agents utilizing InferAct feedback achieved a 10.4% higher success rate than those using standard natural language feedback.

The Advantage of Third-Person Inference

A key finding is that inferring intent from an external perspective is more reliable than agent self-explanation. Agents often "hallucinate" justifications for incorrect actions when asked to explain themselves. Furthermore, InferAct utilizes one-way entailment for verification; an action is deemed safe if the specific inferred intent entails the broader user goal, even if they are not identical.

An example in WebShop where InferAct successfully detects a misalignment that other methods miss.

Limitations and Future Directions

While InferAct significantly enhances agent safety, the authors identify several areas for future research. Currently, "critical actions" must be manually defined, and the system does not distinguish between accidental misalignment and malicious user intent. Future work will focus on automating risk discovery and developing cost-sensitive models that account for the varying impacts of false negatives versus false positives.

Conclusion

InferAct represents a significant step toward the safe deployment of LLM agents in real-world environments. By proactively inferring and verifying agent intent before critical actions are taken, it provides a robust layer of protection against irreversible mistakes while maintaining operational efficiency.

Summary updated

Abstract

A crucial requirement for deploying LLM-based agents in real-life applications is robustness against risky or irreversible mistakes. However, existing research lacks a focus on the preemptive evaluation of reasoning trajectories performed by LLM agents, leading to a gap in ensuring safe and reliable operations. To explore better solutions, this paper introduces InferAct, a novel approach that leverages the Theory-of-Mind capability of LLMs to proactively detect potential errors before critical actions are executed (e.g., "buy-now" in automatic online trading or web shopping). InferAct is also capable of integrating human feedback to prevent irreversible risks and enhance the actor agent's decision-making process. Experiments on three widely used tasks demonstrate the effectiveness of InferAct. The proposed solution presents a novel approach and concrete contributions toward developing LLM agents that can be safely deployed in different environments involving critical decision-making.
BibTeX
@misc{fang2024inferact,
  author = {Haishuo Fang and Xiaodan Zhu and Iryna Gurevych},
  title = {{InferAct: Inferring Safe Actions for LLM-Based Agents Through Preemptive Evaluation and Human Feedback}},
  year = {2024},
  eprint = {2407.11843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.11843},
}
Figure