Normative Testimony and Belief Functions: A Formal Theory of Norm Learning

Taylor Olson, Kenneth D. ForbusIJCAI 2024

Abstract

Normative beliefs are central to human social interaction, yet formalizing how AI agents learn these internal norms from natural language testimony remains a challenge. This paper presents a mathematically grounded framework using Dempster-Shafer belief functions to model the ambiguity and varying reliability of human speakers. By defining a set of essential axioms and a novel fusion rule, the authors provide a sound, computationally efficient method for an agent to infer population-level norms from individual testimony.

Introduction

For AI systems to participate effectively in human society, they must understand our "unwritten rules"—the social and moral norms that dictate what is obligatory, forbidden, or optional. Most existing norm-learning approaches focus on external responses, such as observing social sanctions, or rely on opaque Large Language Models (LLMs) that lack transparent reasoning. This paper addresses the gap in learning internal beliefs from normative testimony (e.g., "You should help others").

Learning from testimony is inherently difficult due to the ambiguity of natural language, the varying reliability of speakers, and the challenge of fusing evidence from multiple sources without allowing outliers to dominate the consensus. The authors propose a formal, axiom-based theory to navigate these complexities.

The Formal Framework

Deontic Logic and Belief Functions

The theory represents norms using the "Three-Fold Classification" (TTC): Obligatory, Impermissible, and Optional. To handle the uncertainty and ignorance inherent in human communication, the authors leverage Dempster-Shafer (DS) Theory rather than traditional Bayesian probability. DS belief functions allow an agent to assign "mass" to sets of possibilities. For instance, if testimony is vague, the agent can assign belief to the set {Obligatory, Optional} (Permissible) without being forced to commit to one specific category.

The "As Reliable as Me" Hypothesis

To evaluate speaker reliability without prior data, the authors introduce a "Theory of Mind" approach. The agent asks: "In this specific context, how likely would I be to tell the truth?" and projects this estimated reliability onto the speaker. This provides a "cold start" mechanism for trust in new environments.

Example of combining normative testimony from two sources using Dempster-Shafer belief functions.

Axiomatic Norm Learning

The core technical contribution is a modified fusion rule for combining evidence. The authors define five essential axioms that any norm-learning function must satisfy:

  1. Conjunctive Pooling: Agreement between sources should strengthen the resulting belief.
  2. Commutativity: The order in which testimony is received should not affect the outcome.
  3. Associativity: The grouping of testimony should not affect the outcome.
  4. Idempotency: Hearing the same testimony twice should not artificially increase certainty.
  5. Non-Dictatorship: No single speaker should be able to override the collective testimony of the population.

Standard DS theory often fails to satisfy idempotency or non-dictatorship in specific contexts. The authors' novel rule ensures all five axioms are met while maintaining deontic consistency—ensuring the AI never concludes an action is simultaneously obligatory and impermissible.

Key Results and Discussion

The paper provides formal proofs for the axiomatic soundness and deontic consistency of the proposed system. Qualitatively, the model was tested using a "running example" of a hypothetical society called the "Flarps." In this simulation, the agent correctly weighed conflicting testimony—such as a reliable friend's advice versus a stressed stranger's exclamation—to accurately infer local norms regarding public behavior.

Crucially, the authors demonstrate that their algorithm has linear complexity relative to the amount of testimony received, making it computationally efficient for real-time applications.

Limitations and Future Work

While the framework is robust, several areas for future research remain. The "As Reliable as Me" hypothesis may struggle if an AI lacks human-like social context. Additionally, the current model assumes testimony independence, which may not hold in "echo chamber" scenarios where speakers repeat one another. Future iterations aim to track individual speaker histories to identify persistent biases and move from descriptive norm learning to more complex social modeling.

Summary updated

BibTeX
@inproceedings{olson2024normative,
  author = {Taylor Olson and Kenneth D. Forbus},
  title = {{Normative Testimony and Belief Functions: A Formal Theory of Norm Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}

Loading related content...

Figure