Assessing Relevance with Extensionally Defined Principles and Cases

AAAI 2000

TL;DR

This paper introduces SIROCCO, a system that models how experts "operationally define" abstract ethical principles by applying them to real-world cases. By representing cases and principles extensionally (through examples and applications), SIROCCO can more accurately predict relevant ethical guidelines and past cases than traditional methods. This work offers a computational approach to understanding and supporting ethical decision-making in complex domains.

What Problem Does This Paper Address?

Many professional domains, like ethics and law, rely on abstract general principles to guide decisions. However, these principles are often too vague to be applied deductively using simple rule-based systems. Experts in these fields don't just apply principles; they interpret and refine them through their decisions over time, effectively creating an "extensional" definition of what the principle means in practice. This paper tackles the challenge of computationally modeling this process to improve the retrieval of relevant principles and past cases for new ethical dilemmas. The authors highlight the difficulty of creating authoritative intermediate rules and the potential for conflicting advice from multiple abstract principles.

Methods

The core of this work is the SIROCCO system, which employs a two-stage retrieval process inspired by case-based reasoning (CBR) techniques.

  • Ethics Transcription Language (ETL): Cases are represented as narratives of chronologically ordered events, including actors, objects, actions, and temporal relationships.
  • Extended ETL (EETL): This extends ETL to capture the analysis of ethics cases, including the BER's conclusion, the protagonist, the argument structure, and specific information about cited code provisions and past cases.
  • Operationalization Techniques: SIROCCO models nine techniques used by experts to "operationally define" principles and cases. The paper focuses on a core subset (techniques 1, 5, 6, 8, and 9) that directly contribute to suggesting relevant cases and codes. These include:
  • Instantiating principles/cases: Linking them to specific facts and temporal sequences.
  • Grouping principles/cases: Organizing them by theme.
  • Reusing past operationalizations: Applying previously established links.
  • Two-Stage Retrieval:
    • Stage 1 (Surface Retrieval): Rapidly matches the target case's "content vectors" (summarizing Fact Chronologies) to those of all source cases, using weighted abstraction levels and focusing on questioned/critical facts.
  • Stage 2 (Structural Mapping): Uses an A* search to find the best structural mapping between the target case and the top-ranked source cases from Stage 1, considering temporal relations and supporting abstract matches.

Datasets

  • Foundational Cases: 184 NSPE BER cases (from 1958-1992) were used to design, implement, and refine SIROCCO. These cases cover a range of important ethics topics.
  • Trial Cases: 58 cases decided later than the foundational cases (1993-1998) were used for evaluation. These were divided into "Selected Topics" (44 cases) and "Non-Selected Topics" (14 cases).

Key Insights and Ideas

  • Extensional Definition of Principles: The paper provides empirical evidence that experts' explanations of ethical principles effectively "operationally define" them. This means that the meaning and applicability of abstract principles are built up through concrete examples and applications over time.
  • Value of Operationalization: SIROCCO's success demonstrates that computationally modeling these operationalization techniques is crucial for accurate retrieval of relevant ethical guidance. Simply matching keywords or abstract concepts is insufficient.
  • Narrative Case Representation: Using a detailed, temporal narrative representation (ETL) allows for a richer understanding of case facts and their relationships, which is vital for ethical reasoning.
  • Surprising Temporal Finding: The authors were surprised that temporal knowledge did not significantly improve SIROCCO's accuracy. They hypothesize this might be because temporal orderings leading to different ethical interpretations are rare in the NSPE BER case corpus.

Key Results

  • SIROCCO significantly outperformed five competitor methods (RANDOM, INFORMED-RANDOM, MG, EXTENDED-MG, and NON-OP SIROCCO) in predicting relevant codes and cases, as measured by the F-Measure.
  • The difference in accuracy between SIROCCO and EXTENDED-MG (a strong full-text retrieval method) was statistically significant for exact matching and showed a strong trend for inexact matching.
  • A supplemental experiment where ethics graduate students evaluated extra suggestions indicated that SIROCCO provided more reasonable additional citations than EXTENDED-MG.
  • SIROCCO's predictive capability degraded slightly when applied to cases outside its primary "Selected Topics" area, but it still performed well.
  • An ablated version of SIROCCO (NON-TEMP SIROCCO) that did not use temporal knowledge showed essentially negligible differences in performance, supporting the authors' hypothesis about the rarity of critical temporal distinctions in their dataset.

Limitations and Future Work

  • Temporal Knowledge Contribution: The authors acknowledge that their finding that temporal knowledge didn't significantly improve accuracy was surprising and may be due to the specific dataset. They suggest that in other domains or with different case selection, temporal aspects might be more critical.
  • Effort of Representation: Representing cases in ETL is more time-consuming than encoding input for full-text retrieval systems.
  • Future Work:
    • Incorporating SIROCCO into a tutoring environment for practical ethics, leveraging the pedagogical value of problem representation.
  • Exploring how CBR knowledge representations can guide automated index assignment for textual cases.
  • Investigating the potential for temporal knowledge to be more impactful in other domains.

Summary updated

BibTeX
@inproceedings{2000assessing,
  title = {{Assessing Relevance with Extensionally Defined Principles and Cases}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2000},
}
Figure