Foveate, Attribute, and Rationalize: Towards Safe and Trustworthy AI

Alex Mei, Sharon Levy, William Yang Wang

TL;DR

This paper introduces FARM, a novel framework that tackles the challenge of identifying and explaining "covertly unsafe" text – language that implies physical harm but requires reasoning to detect. FARM leverages external knowledge, attributing it to trustworthy sources, to generate human-interpretable rationales, significantly improving AI safety classification accuracy.

What Problem Does This Paper Address?

As intelligent systems become more integrated into our lives, ensuring user safety is paramount. A growing concern is "covertly unsafe text," which, unlike explicitly violent content, subtly suggests dangerous actions that could lead to serious injury. This type of text is particularly challenging to detect because it often arises from everyday scenarios and requires contextual understanding and external knowledge to identify the inherent risks. Current AI safety research has largely focused on explicitly harmful content, leaving a critical gap in addressing these more insidious forms of unsafe language. The authors aim to build AI systems that can not only identify this covertly unsafe text but also provide clear, trustworthy explanations for their decisions, empowering users and stakeholders to better manage risks.

Methods

The FARM (Foveate, Attribute, Rationalize) methodology is a three-step pipeline designed to generate trustworthy rationales for safety scenarios:

  1. Foveation: This novel step uses few-shot prompting to guide large language models (LLMs) to identify the specific pieces of external knowledge needed to understand a given safety scenario. It's inspired by the human visual process of foveation, focusing on the most relevant information to denoise the query and improve interpretability.
  2. Attribution: The identified "foveations" are then used to query external knowledge sources. The key innovation here is the explicit attribution of retrieved information to trustworthy sources (like Wikipedia, Google Base, and Google Credible) to minimize misinformation and allow for fact-checking.
  3. Rationalization: Finally, the attributed external knowledge is combined with the original scenario to generate human-interpretable rationales. These rationales explain why a particular action is deemed safe or unsafe, providing a clear justification.

Datasets

The primary dataset used for evaluation is SAFETEXT, a benchmark specifically designed for covertly unsafe text. The authors also mention augmenting this dataset with human-interpretable rationales.

Key Insights and Ideas

  • Foveation as a Denoising Mechanism: The idea of explicitly prompting LLMs to identify what knowledge is missing, rather than just asking for information, is a clever way to focus the retrieval process and improve the signal-to-noise ratio.
  • Trustworthy Attribution is Crucial for Safety: Simply retrieving external knowledge isn't enough; ensuring that knowledge comes from reputable sources is vital to prevent the AI from generating misinformation, which could be even more dangerous in safety-critical contexts.
  • Human-Interpretable Rationales Build Trust: The ultimate goal is not just classification but also explanation. Providing clear, attributed rationales allows users to understand why an AI made a decision, fostering trust and enabling them to verify the information themselves.
  • Pipelined Approach Mitigates Complexity: Breaking down the problem into foveation, attribution, and rationalization allows each step to be less challenging individually, potentially leading to better overall performance.

Key Results

FARM achieved state-of-the-art results on the SAFETEXT dataset, demonstrating a significant improvement in safety classification accuracy. Specifically, FARM showed an absolute improvement of 5.9% in overall safety classification accuracy compared to the previous baseline. The paper also highlights improvements in the factuality, entailment, and faithfulness of the generated rationales.

Limitations and Future Work

  • External Knowledge Volatility: The accuracy of FARM relies on the up-to-dateness of external knowledge sources. As information changes, ensuring the system always accesses the most current and correct data is a challenge.
  • Reasoning Model Dependence: FARM's performance is tied to the capabilities of the underlying LLMs used for foveation, rationalization, and classification. Advances in LLMs will likely further enhance FARM's effectiveness.
  • Prompt Engineering: While few-shot prompting is effective, finding the optimal prompts for different models and tasks may require significant tuning.
  • Future Work: The authors suggest applying FARM to other domains beyond AI safety, such as legal or medical contexts, where accurate and attributed knowledge is critical. They also emphasize the need for interdisciplinary solutions to AI safety, including user education on verifying attributed sources.

Summary updated

Abstract

Users' physical safety is an increasing concern as the market for intelligent systems continues to grow, where unconstrained systems may recommend users dangerous actions that can lead to serious injury. Covertly unsafe text is an area of particular interest, as such text may arise from everyday scenarios and are challenging to detect as harmful. We propose FARM, a novel framework leveraging external knowledge for trustworthy rationale generation in the context of safety. In particular, FARM foveates on missing knowledge to qualify the information required to reason in specific scenarios and retrieves this information with attribution to trustworthy sources. This knowledge is used to both classify the safety of the original text and generate human-interpretable rationales, shedding light on the risk of systems to specific user groups and helping both stakeholders manage the risks of their systems and policymakers to provide concrete safeguards for consumer safety. Our experiments show that FARM obtains state-of-the-art results on the SafeText dataset, showing absolute improvement in safety classification accuracy by 5.9%.
BibTeX
@inproceedings{mei2023foveate,
  author = {Alex Mei and Sharon Levy and William Yang Wang},
  title = {{Foveate, Attribute, and Rationalize: Towards Safe and Trustworthy AI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=VjExLs5KEg_},
}
Figure