Is There a Role for Qualitative Risk Assessment?

Paul J. Krause, John Fox, Philip N. JudsonUAI 1995

TL;DR

This paper argues that traditional quantitative risk assessment methods are insufficient for domains like chemical carcinogenicity, where data is sparse and uncertain. The authors propose extending risk assessment to incorporate qualitative and "weakly quantitative" approaches, focusing on a computer-based system that uses argumentation to represent and communicate risk.

What Problem Does This Paper Address?

The paper tackles the challenge of assessing risk in situations where precise, objective numerical probabilities are difficult or impossible to obtain. This is particularly relevant in fields like chemical carcinogenicity, where only a small fraction of known substances have been tested, and extrapolating from animal studies to humans is fraught with uncertainty. The authors highlight the disconnect between "objective" risk estimates (often based on limited data) and "subjective" or perceived risk, which can lead to societal conflict and ineffective decision-making. They argue that traditional quantitative methods, which rely on point-value probabilities, fail to adequately capture the inherent uncertainties and can lead to misleading conclusions when data is sparse.

Methods

The core of the paper introduces the StAR (Standardised Argument Report) project, which aims to develop a computer-based assistant for risk assessment. The key innovation lies in its argumentation-based approach. Instead of relying solely on numerical probabilities, StAR constructs arguments "for" and "against" a particular risk hypothesis. These arguments are built from facts and rules, and their strength is evaluated.

The paper proposes a framework for representing these arguments and "cases" (collections of arguments impacting a proposition). They introduce a set of qualitative linguistic terms (e.g., "supported," "plausible," "probable," "confirmed," "certain") that are defined not by probabilistic intervals, but by the structure and consistency of the underlying arguments. This allows for a more nuanced representation of uncertainty, moving beyond simple numerical values. The system uses a prototype demonstrator that analyzes chemical structures for potential carcinogenic hazards based on known "structural alerts" and generates reports that summarize the arguments for and against carcinogenicity.

Datasets

Not discussed in the paper. The paper focuses on the methodology and a prototype system, using simplified examples for illustration rather than large-scale datasets.

Key Insights and Ideas

  • The limitations of purely quantitative risk assessment: In domains with sparse and uncertain data (like chemical carcinogenicity), point-value probabilities can be misleading and fail to capture the full picture of risk.
  • The importance of argumentation: Representing risk as a balance of arguments for and against a hypothesis provides a more transparent and understandable way to communicate uncertainty.
  • "Weakly quantitative" and qualitative risk characterization: Moving beyond strict numerical values to incorporate linguistic terms that reflect the strength and nature of evidence is crucial for effective risk communication.
  • Cognitive validity of linguistic terms: The authors emphasize the need for qualitative terms to align with how people intuitively understand and use them, suggesting experimental work to validate these associations.
  • Transparency in risk communication: The argumentation framework aims to make the reasoning behind a risk assessment explicit, allowing recipients to understand the justification for a conclusion and act as final arbiters.

Key Results

The paper demonstrates a prototype system (StAR) that can analyze chemical structures and generate "equivocal" or "improbable" risk reports based on the balance of arguments. For example, one report concluded that the case for a substance being carcinogenic was "equivocal" due to conflicting evidence regarding absorption and metabolic activation. While specific quantitative performance metrics (like accuracy) are not presented, the qualitative output showcases the system's ability to articulate nuanced risk assessments based on structured arguments. The authors suggest that the linguistic terms used in the reports have a correspondence with underlying measures of belief and plausibility, though they are not strictly defined by fixed probability intervals.

Limitations and Future Work

The authors acknowledge several limitations and areas for future work:

  • Cognitive validity of linguistic terms: It remains an open question whether the proposed qualitative terms have "cognitive validity" and will be interpreted as intended by users. Experimental work is underway to address this.
  • Decision-making framework: While the system provides a risk characterization, a clear decision rule for acting on this qualitative assessment still needs to be developed.
  • Numerical coefficients: The numerical coefficients used in the demonstrator were somewhat arbitrary and intended for illustration. Future work will explore how to realistically assign and interpret these values, potentially as measures of plausibility rather than strict probabilities.
  • Modeling generic knowledge: The authors suggest that modeling generic knowledge as conditional assertions (e.g., using possibilistic logic) might be more appropriate than the current rule-based approach with certainty coefficients.
  • Scope of argumentation: The paper presents a selection of argument structures, and further research is needed to identify which landmarks are most important and how to consistently associate them with natural language terms.

Summary updated

BibTeX
@inproceedings{krause1995is,
  author = {Paul J. Krause and John Fox and Philip N. Judson},
  title = {{Is There a Role for Qualitative Risk Assessment?}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1995},
}
Figure