Survey of Trustworthy AI: A Meta Decision of AI

Caesar Wu, Yuan-Fang Li, Pascal BouvryarXiv 2023 · OpenReview

Abstract: When making strategic decisions, we are often confronted with overwhelming information to process. The situation can be further complicated when some pieces of evidence are contradicted each other or paradoxical. The challenge then becomes how to determine which information is useful and which ones should be eliminated. This process is known as meta-decision. Likewise, when it comes to using Artificial Intelligence (AI) systems for strategic decision-making, placing trust in the AI itself becomes a meta-decision, given that many AI systems are viewed as opaque "black boxes" that process large amounts of data. Trusting an opaque system involves deciding on the level of Trustworthy AI (TAI). We propose a new approach to address this issue by introducing a novel taxonomy or framework of TAI, which encompasses three crucial domains: articulate, authentic, and basic for different levels of trust. To underpin these domains, we create ten dimensions to measure trust: explainability/transparency, fairness/diversity, generalizability, privacy, data governance, safety/robustness, accountability, reproducibility, reliability, and sustainability. We aim to use this taxonomy to conduct a comprehensive survey and explore different TAI approaches from a strategic decision-making perspective.

Introduction: Trust as a Meta-Decision

As Artificial Intelligence (AI) transitions from experimental labs to high-stakes strategic environments—such as healthcare, financial markets, and judicial systems—the "black box" nature of deep learning has created a significant crisis of confidence. This paper frames the act of trusting AI not as a binary technical requirement, but as a meta-decision: the strategic process of determining how and when to rely on an opaque system when faced with contradictory or paradoxical evidence.

Strategic decision-makers often face the "AI Effect Paradox": once an AI technique becomes common and reliable, it is no longer perceived as "AI" but as standard technology, shifting the threshold for what constitutes "trustworthiness." To navigate this, a unified framework is required to evaluate AI through the lenses of justice, security, and dependence.

A Unified Taxonomy for Trustworthy AI

The core contribution of this research is a hierarchical taxonomy designed to bridge the gap between abstract ethical principles and concrete technical implementation. This framework organizes the complex landscape of TAI into four distinct layers:

  1. Strategic Objectives: The high-level goals of the organization, such as revenue growth, innovation, or risk management.
  2. Domains of Knowledge: Three levels of trust—Articulate (explicitly defined), Authentic (balanced risk), and Basic (baseline norms).
  3. Dimensions of Measurement: Ten categories that allow for the quantification of trust, including Explainability, Fairness, and Sustainability.
  4. Implementation Models & Schemes: The specific technical tools used to achieve these dimensions.
The hierarchical framework of Trustworthy AI, mapping strategic objectives to domains of trust and ethical lenses.

This structure is underpinned by three primary ethical theories: Utilitarianism (utility-based), Deontology (duty-based), and Natural Law (virtue-based). By aligning technical dimensions with these philosophical foundations, decision-makers can ensure that AI systems not only perform efficiently but also align with human flourishing and societal duties.

Mapping the TAI Toolbox

To move from theory to practice, the authors conducted a systematic literature review following the PRISMA methodology, screening over 1,000 papers to identify seminal works. This resulted in a comprehensive mapping of 33 implementation models and 106 specific technical schemes to the ten dimensions of trust.

A detailed taxonomy mapping the 10 dimensions of trust to 33 implementation models and 106 technical schemes/tools.

For example, the dimension of Fairness is addressed through implementation models like "Fair Learning" and tools such as IBM’s AIF360, which offers pre-processing and post-processing algorithms to mitigate bias. Similarly, Privacy is supported by models like Federated Learning (FL) and schemes like Differential Privacy (DP). This mapping provides engineers and decision-makers with a "toolbox" to address specific technical challenges while maintaining a holistic view of trust.

The Reproducibility Crisis and Strategic Paradoxes

A critical finding of the paper’s meta-analysis is the ongoing reproducibility crisis in AI research. Analyzing 1,726 papers from top-tier conferences (IJCAI and AAAI) between 2013 and 2016, the authors found that zero papers in their sample were fully reproducible (R1 level, providing both software and data). While the inclusion of pseudocode has improved, the lack of shared datasets and experimental environments remains a significant barrier to "Basic Trust."

Furthermore, the paper identifies over 20 "strategic paradoxes" that complicate TAI. A primary example is the Transparency Paradox: while increasing explainability (XAI) can build trust, it can also make a system more vulnerable to adversarial attacks by revealing the underlying logic to bad actors. Similarly, the Accuracy Paradox suggests that near-perfect accuracy (100%) is often a red flag for overfitting or imbalanced data, which actually decreases real-world reliability.

Conclusion

Trustworthy AI is not a destination but a continuous meta-decision process. While the proposed taxonomy provides a robust technical and ethical roadmap, the authors acknowledge that trust remains partially emotional and relational—elements that are difficult to quantify. Future work will focus on implementing this framework within High-Performance Computing (HPC) environments to transform these abstract ethical paradoxes into solvable computational problems that align with specific business objectives.

Summary updated

Abstract

When making strategic decisions, we are often confronted with overwhelming information to process. The situation can be further complicated when some pieces of evidence are contradicted each other or paradoxical. The challenge then becomes how to determine which information is useful and which ones should be eliminated. This process is known as meta-decision. Likewise, when it comes to using Artificial Intelligence (AI) systems for strategic decision-making, placing trust in the AI itself becomes a meta-decision, given that many AI systems are viewed as opaque "black boxes" that process large amounts of data. Trusting an opaque system involves deciding on the level of Trustworthy AI (TAI). We propose a new approach to address this issue by introducing a novel taxonomy or framework of TAI, which encompasses three crucial domains: articulate, authentic, and basic for different levels of trust. To underpin these domains, we create ten dimensions to measure trust: explainability/transparency, fairness/diversity, generalizability, privacy, data governance, safety/robustness, accountability, reproducibility, reliability, and sustainability. We aim to use this taxonomy to conduct a comprehensive survey and explore different TAI approaches from a strategic decision-making perspective.
BibTeX
@misc{wu2023survey,
  author = {Caesar Wu and Yuan-Fang Li and Pascal Bouvry},
  title = {{Survey of Trustworthy AI: A Meta Decision of AI}},
  year = {2023},
  eprint = {2306.00380},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.00380},
}
Figure