TL;DR
This paper introduces a comprehensive framework for evaluating the trustworthiness of Generative Foundation Models (GenFMs). It proposes a set of guiding principles derived from global AI governance and industry standards, and presents TRUSTGEN, a novel dynamic benchmarking platform for assessing GenFMs across multiple dimensions like truthfulness, safety, and fairness. The work highlights significant progress in GenFM trustworthiness but also identifies persistent challenges, advocating for a holistic approach to ensure responsible AI deployment.
What Problem Does This Paper Address?
The widespread adoption of Generative Foundation Models (GenFMs) has brought about transformative advancements, but it also raises critical concerns about their trustworthiness. These concerns span various dimensions, including truthfulness, safety, fairness, robustness, and privacy. Existing evaluation methods are often static and struggle to keep pace with the rapid evolution of GenFMs and the diverse needs of downstream applications. This paper addresses the gap by proposing a unified framework for defining, assessing, and guiding the enhancement of GenFM trustworthiness, aiming to ensure their safe and responsible integration into critical applications.
Methods
The paper's contributions are threefold:
- Guiding Principles: A systematic review of global AI governance laws, policies, and industry standards was conducted. This analysis informed the development of a set of guiding principles for GenFMs, integrating technical, ethical, legal, and societal perspectives. These principles are designed to be adaptable and serve as a foundational reference for GenFM development, evaluation, and governance.
- TRUSTGEN Benchmark: The authors introduce TRUSTGEN, a dynamic benchmarking platform designed to evaluate GenFM trustworthiness across multiple dimensions (truthfulness, safety, fairness, robustness, privacy, machine ethics, and advanced AI risk) and model types (text-to-image, large language, and vision-language models). TRUSTGEN employs modular components:
- Metadata Curator: Gathers relevant metadata.
- Test Case Builder: Generates test cases based on metadata.
-
Contextual Variator: Ensures test cases are varied and representative across different contexts to mitigate prompt sensitivity. This dynamic approach allows for adaptive and iterative assessments, overcoming the limitations of static methods.
-
In-depth Discussion: The paper provides a comprehensive discussion on the challenges and future directions for trustworthy GenFMs, exploring the evolving nature of trustworthiness, trade-offs between utility and trustworthiness, and considerations for downstream applications.
Datasets
The paper utilizes several datasets for evaluation and benchmark construction:
- Existing Datasets: The authors leverage existing datasets for various tasks, including:
- CrowS-Pairs, StereoSet, and BBQ for stereotype and disparagement evaluation.
- TruthfulQA for truthfulness evaluation.
- VISPR and Vizwiz_Priv for privacy evaluation.
- Social-Chemistry-101, MoralChoice, Ethics, NormBank, and Moral Stories for machine ethics evaluation.
- HallusionBench and AutoHallusion for VLM truthfulness/hallucination evaluation.
- Various datasets for LLM robustness evaluation, including annotated datasets with ground-truth labels and open-ended question-answering datasets.
- Datasets for cybersecurity applications, such as those related to prompt injection attacks.
- Newly Introduced Benchmarks/Frameworks:
- TRUSTGEN: A novel dynamic benchmarking platform for GenFMs.
- TRUSTEVAL Toolkit: An open-source toolkit released to facilitate dynamic evaluation.
- MMBias: A benchmark of 3,800 images and phrases across 14 population subgroups for bias assessment in VLMs.
- AutoDetect: A unified framework for automatically uncovering LLM flaws.
- JailbreakV-28K: A benchmark for assessing the robustness of multimodal large language models against jailbreak attacks.
- MMJ-Bench: A standardized and comprehensive evaluation of existing VLM jailbreak attack and defense techniques.
- Retention Score: A metric to quantify jailbreak risks of VLMs using diffusion models.
- AI RMF 1.0: A framework for building AI systems responsibly.
- AutoTrust: A comprehensive trustworthiness benchmark for MLLMs for driving.
- MMSafetyBench: A benchmark for safety evaluation of multimodal large language models.
- MMDU: A multi-turn multimodal dialogue understanding benchmark and instruction-tuning dataset for VLMs.
- VLM-as-a-Judge: An LLM-based evaluation method.
Key Insights and Ideas
- Trustworthiness is Dynamic and Context-Dependent: The paper emphasizes that trustworthiness is not a static attribute but an evolving quality that varies significantly across different downstream applications, stakeholders, and cultural contexts. A one-size-fits-all approach is insufficient.
- Holistic Framework is Crucial: Addressing GenFM trustworthiness requires a comprehensive framework that integrates technical, ethical, legal, and societal perspectives. Improvements in one dimension can impact others, necessitating a holistic approach.
- Dynamic Benchmarking is Essential: Static evaluation methods are insufficient for the rapidly evolving GenFM landscape. Dynamic benchmarks like TRUSTGEN, which adapt to evolving ethical standards and model capabilities, are crucial for continuous and reliable assessment.
- Open-Source Models are Competitive: Contrary to common perceptions, open-source models are increasingly matching or even surpassing the performance of frontier proprietary models in trustworthiness metrics.
- Utility and Trustworthiness are Intertwined: Enhancing trustworthiness should not come at the cost of utility. The paper argues for approaches that balance both, suggesting that harmlessness can serve as a foundation for maximizing utility.
- Ambiguities in Safety Assessment: The paper highlights the challenges in distinguishing between harmful and benign content, both in inputs and outputs, due to differing interpretations and the potential for subtle manipulation.
- Interdisciplinary Collaboration is Key: Ensuring GenFM trustworthiness requires collaboration across diverse fields, including AI development, ethics, law, social sciences, and domain-specific expertise (e.g., medicine, robotics).
Key Results
The paper presents several key findings from their evaluation using TRUSTGEN:
- Overall Performance: State-of-the-art GenFMs generally perform well but still face "trustworthiness bottlenecks."
- Text-to-Image Models: Generally perform well but fall short in generating complex scenes with multiple objects and interrelationships. Dall-E 3 shows the best performance in truthfulness, while HunyuanDiT leads in fairness. Privacy leakage rates vary significantly, with Claude-3-Haiku and Llama-3.2-11B-V showing strong performance.
- Large Language Models: Exhibit significant variability in sycophancy levels. Models perform poorly in "Self-Identity Cognition" but excel in "Latest Information with External Services." Toxicity levels are generally low across models, with no substantial difference between open-source and proprietary models. Exaggerated safety performance is strong, with most models demonstrating low refusal rates.
- Vision-Language Models: Show varying levels of robustness across tasks, with higher robustness on VQA compared to image captioning. GPT-40 and Claude-3.5-Sonnet are top performers in truthfulness. Gemini-1.5-Pro achieves high accuracy in stereotype and disparagement tasks, while Llama-3.2-90B-V shows low accuracy. Models generally maintain neutrality in preference tasks, with Mixtral-8x22B achieving outstanding performance.
- Advanced AI Risks: Larger and more advanced language models generally outperform smaller or earlier models in assessing advanced AI risks. QwQ-32B and Gemma-2-27B show strong performance.
Limitations and Future Work
Limitations:
- Static Benchmarks: The paper acknowledges the limitations of static evaluation methods and the need for dynamic approaches.
- Prompt Sensitivity: While addressed by the Contextual Variator, prompt sensitivity remains a challenge in model evaluation.
- Data Contamination: The potential for benchmark leakage or cheating by models is a concern, which dynamic evaluation aims to mitigate.
- Ambiguities in Safety: The paper highlights the inherent ambiguities in defining and assessing harmful content, both in inputs and outputs.
- Model-Specific Performance: While general trends are observed, specific models may exhibit unique strengths and weaknesses across different trustworthiness dimensions.
Future Work:
- Continuous Evaluation: The dynamic nature of TRUSTGEN allows for continuous updates to datasets and evaluation metrics, ensuring its relevance as GenFMs evolve.
- Expanding Model Coverage: The framework is designed to accommodate new model types and architectures.
- Interdisciplinary Integration: The paper emphasizes the need for continued collaboration across disciplines to refine ethical frameworks and trustworthiness criteria.
- Addressing Advanced AI Risks: The paper calls for a new paradigm for governing GenFMs, focusing on emergent advanced AI risks like self-replication, persuasion, and anthropomorphism.
- Refining Definitions: The need for clearer definitions of trustworthiness dimensions, particularly for safety and fairness, is highlighted.
- Balancing Utility and Trustworthiness: Future research should focus on approaches that enhance both utility and trustworthiness simultaneously, rather than treating them as a trade-off.
Summary updated