Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI

Alex Mei, Sharon Levy, William Yang Wang

Introduction

As Large Language Models (LLMs) become increasingly integrated into daily life, ensuring their safety is paramount. While significant research has focused on "overtly unsafe" content—such as hate speech or explicit violence—a more subtle threat exists: covertly unsafe text. This category includes advice that appears benign on the surface but can lead to physical harm if followed. For instance, suggesting a child "store lemonade in a copper vessel" or "try the cinnamon challenge" requires specific chemical or physiological knowledge to identify as dangerous.

Standard LLMs often fail in these scenarios because their internal knowledge is static and they are prone to hallucinations. To address this, researchers at UC Santa Barbara have developed FARM (Foveate, Attribute, and Rationalize), a framework designed to ground AI reasoning in verifiable external knowledge.

The FARM Framework

The FARM pipeline mimics human cognitive processes to evaluate safety through three distinct stages:

  1. Foveate: Much like the human eye focuses on a specific point to gain clarity, the model first identifies what it doesn't know. It generates a targeted query to find the missing context (e.g., "interaction between lemonade and a copper vessel").
  2. Attribute: The system uses these queries to retrieve information from trustworthy external sources. The researchers utilized Wikipedia and a filtered Google search (restricted to .gov, .edu, and .org domains) to ensure the credibility of the retrieved snippets.
  3. Rationalize: In the final stage, the model synthesizes the original input with the retrieved evidence to produce a safety classification and a human-readable explanation.
The FARM pipeline: Foveation identifies missing context, Attribution retrieves external facts, and Rationalization generates a safe/unsafe decision with an explanation.

Experimental Evaluation

The researchers evaluated FARM using the SAFETEXT dataset, which contains 1,465 examples of everyday scenarios categorized as safe or unsafe. The team augmented this dataset with human-evaluated foveations and rationales to provide a robust benchmark for physical safety.

Key Results

The implementation of the FARM framework led to a significant performance leap over existing baselines.

Performance comparison of the FARM framework against the SAFETEXT baseline across different knowledge sources.

As shown in the results table, FARM achieved a state-of-the-art 91.4% overall accuracy, representing a 5.9% absolute improvement over the SAFETEXT baseline. The most notable gains were in the "Unsafe" subset, where the model's ability to detect dangerous advice improved by nearly 20% in certain configurations. The "Credible-1" and "Credible-3" configurations (using filtered Google results) proved most effective, highlighting that the quality of external knowledge is a critical factor in safety performance.

Insights and Discussion

The study revealed several critical insights into the behavior of grounded LLMs:

  • The Conservative Shift: Access to external knowledge makes models more cautious. When presented with evidence of potential harm, models are significantly more likely to classify borderline actions as unsafe, effectively reducing dangerous false negatives.
  • The Foveation Bottleneck: The primary challenge in the pipeline is the initial "foveation" step. If the model fails to identify the correct knowledge gap, the subsequent retrieval and reasoning steps are compromised. Larger models like Davinci-003 showed significantly better foveation capabilities than smaller counterparts.
  • Mitigating Hallucination: By requiring the model to attribute its reasoning to specific retrieved snippets, FARM drastically reduces "source hallucination," where models invent citations to justify incorrect safety claims.

Conclusion and Future Work

The FARM framework demonstrates that grounding LLMs in external, credible knowledge is a viable path toward safer AI. While currently focused on physical safety, the authors suggest that this "Foveate-Attribute-Rationalize" approach could be extended to other high-stakes fields such as legal advice, medical diagnostics, and complex technical reasoning. Future work will focus on improving the foveation step and ensuring the system remains robust as new safety risks emerge in the real world.

Summary updated

Abstract

Users' physical safety is an increasing concern as the market for intelligent systems continues to grow, where unconstrained systems may recommend users dangerous actions that can lead to serious injury. Covertly unsafe text is an area of particular interest, as such text may arise from everyday scenarios and are challenging to detect as harmful. We propose FARM, a novel framework leveraging external knowledge for trustworthy rationale generation in the context of safety. In particular, FARM foveates on missing knowledge to qualify the information required to reason in specific scenarios and retrieves this information with attribution to trustworthy sources. This knowledge is used to both classify the safety of the original text and generate human-interpretable rationales, shedding light on the risk of systems to specific user groups and helping both stakeholders manage the risks of their systems and policymakers to provide concrete safeguards for consumer safety. Our experiments show that FARM obtains state-of-the-art results on the SafeText dataset, showing absolute improvement in safety classification accuracy by 5.9%.
BibTeX
@inproceedings{mei2023foveate,
  author = {Alex Mei and Sharon Levy and William Yang Wang},
  title = {{Foveate, Attribute, and Rationalize: Towards Physically Safe and Trustworthy AI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=VjExLs5KEg_},
}
Figure