Establishing Meta-Decision-Making for AI: An Ontology of Relevance, Representation and Reasoning

Cosmin Badea, Leilani GilpinarXiv 2022 · OpenReview

Abstract

We propose an ontology of building decision-making systems, with the aim of establishing Meta-Decision-Making for Artificial Intelligence (AI), improving autonomy, and creating a framework to build metrics and benchmarks upon. To this end, we propose the three parts of Relevance, Representation, and Reasoning, and discuss their value in ensuring safety and mitigating risk in the context of third-wave cognitive systems. Our nomenclature reflects the literature on decision-making, and our ontology allows researchers that adopt it to frame their work in relation to one or more of these parts.

Introduction

As artificial intelligence moves toward what DARPA defines as the "Third Wave"—systems characterized by context-awareness and the ability to adapt to novel situations—the mechanisms by which these systems make decisions must evolve. Current automated decision-making systems, such as those used in autonomous vehicles or medical diagnostics, often function as rigid rule-followers or opaque "black boxes." When these systems encounter "corner cases"—unforeseen scenarios absent from their training data—they frequently fail in ways that are difficult to audit or correct.

To address this, we propose a shift from simple decision-making to Meta-Decision-Making: the process of reasoning about how decisions are made. This paper introduces a structured ontology designed to provide a common nomenclature for researchers, allowing for better auditing, benchmarking, and risk mitigation in autonomous systems.

The Meta-Decision-Making Gap

The primary challenge in modern AI is the lack of a structured framework to decompose the decision-making pipeline. Without a clear ontology, it is difficult to determine why a system failed. Was the failure due to a perception error, a misunderstanding of the rules, or a flaw in the logic of the action? By establishing a meta-level understanding of these processes, we can move away from brittle, isolated error handling toward systems that can adapt their decision-making strategies to complex, real-world contexts.

The Three R’s Ontology

Our framework categorizes the decision-making process into three distinct, interacting stages: Relevance, Representation, and Reasoning.

1. Relevance (Identifying Saliency)

Relevance serves as the "perception" layer of the agent. In any given environment, an agent is bombarded with data; the Relevance stage involves determining which environmental factors are salient to the current goal. For instance, in an autonomous driving context, the presence of a pedestrian is highly relevant, while the specific shape of a distant cloud is not. The goal of this stage is to extract saliency from context, ensuring the system focuses its computational resources on the factors that truly matter.

2. Representation (Syntax and Semantics)

Once relevant factors are identified, they must be translated into a technical form—whether through logic, symbols, or machine learning features. This stage addresses the "Interpretation Problem." It is not enough to provide an AI with a set of rules; the system must have a robust way to encode the meaning of those rules. Representation ensures that the syntax used by the machine aligns with the intended semantics of the human designers, bridging the gap between raw data and actionable information.

3. Reasoning (Making the Decision)

Reasoning is the final step where the agent utilizes represented data to select an action. Our ontology distinguishes between two primary modes of reasoning:

  • Normative Reasoning: How a rational agent should act based on established rules or logic.
  • Descriptive Reasoning: How agents (often humans) actually act in real-world scenarios.

A meta-decision-making system must navigate the tension between these modes, maintaining enough flexibility to avoid "decision paralysis" when rules conflict or when faced with unprecedented scenarios.

Safety, Risk Mitigation, and Autonomy

By modularizing the decision-making process, the Three R’s framework provides a clear roadmap for safety auditing. If a failure occurs, developers can pinpoint the specific "R" responsible:

  • Relevance Failure: The system omitted a critical factor.
  • Representation Failure: The system misinterpreted a rule or priority.
  • Reasoning Failure: The system failed to calculate the optimal path or suffered from rule explosion.

Using the MARS (a value-based moral decision-making system) as a conceptual example, we demonstrate that redundancy is a feature of meta-decision-making. If one reasoning framework (e.g., a specific symbolic logic) proves inadequate for a new environment, a meta-autonomous agent should be able to shift to an alternative reasoning model. True autonomy is achieved when an agent can handle these three steps internally, rather than relying on manual tuning by human programmers for every new context.

Conclusion and Future Work

This ontology provides the foundational nomenclature necessary for the next generation of AI research. While currently a conceptual framework, the Three R's allow for the future development of empirical metrics and benchmarks to audit AI safety.

Future work will focus on the "Interpretation Problem" and the challenge of "Rule Explosion," where increasing complexity leads to computational bottlenecks. Additionally, the transition from manual, human-led relevance identification to fully automated saliency detection remains a significant hurdle. By adopting this ontology, the AI community can better align technical development with the philosophical and safety requirements of truly autonomous systems.

Summary updated

Abstract

We propose an ontology of building decision-making systems, with the aim of establishing Meta-Decision-Making for Artificial Intelligence (AI), improving autonomy, and creating a framework to build metrics and benchmarks upon. To this end, we propose the three parts of Relevance, Representation, and Reasoning, and discuss their value in ensuring safety and mitigating risk in the context of third wave cognitive systems. Our nomenclature reflects the literature on decision-making, and our ontology allows researchers that adopt it to frame their work in relation to one or more of these parts.
BibTeX
@misc{badea2022establishing,
  author = {Cosmin Badea and Leilani Gilpin},
  title = {{Establishing Meta-Decision-Making for AI: An Ontology of Relevance, Representation and Reasoning}},
  year = {2022},
  eprint = {2210.00608},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.00608},
}
Figure