Avoiding Undertrust and Overtrust

Hermann Kaindl, Davor SvetinovicREFSQ Workshops 2019 · OpenReview

TL;DR

This paper argues that for safety-critical systems, simply ensuring actual safety isn't enough; users' perceived safety must also be managed. The authors propose learning from Human Factors research to balance undertrust and overtrust, highlighting the need for new requirements engineering approaches to achieve this equilibrium.

What Problem Does This Paper Address?

The paper tackles a critical challenge in the development of safety-critical systems: how to ensure users appropriately trust these systems. Even if a system is demonstrably safe according to objective measures, users might not trust it (undertrust), leading to them overriding its functions, or they might trust it too much (overtrust), leading to complacency and potential accidents. This is particularly relevant for increasingly autonomous systems like self-driving cars. The authors point out that trust is a complex psychological and sociological concept, not just a technical property, and current requirements engineering practices often overlook this nuance. They aim to bridge the gap between technical system safety and human perception of that safety.

Methods

The paper doesn't present a novel algorithm or a specific experimental methodology. Instead, its primary "method" is a call to action and a conceptual framework. The authors propose to:

  • Learn from Human Factors: They advocate for drawing insights from the established field of Human Factors, which studies the interaction between humans and technical systems, focusing on reducing errors, increasing productivity, and enhancing safety and comfort.
  • Integrate Social Science Research: They suggest incorporating standard social science research methods (case studies, surveys, observational, correlational, experimental, and cross-cultural methods) into requirements engineering.
  • Reframe Engineering Questions: A key methodological proposal is to phrase engineering questions in a way that can be effectively addressed by social science research, and to find methods to engineer the results of these studies back into the technical systems.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Trust is a Two-Sided Coin: The paper emphasizes that both undertrust (perceived safety < actual safety) and overtrust (perceived safety > actual safety) are detrimental. The ideal state is perceived safety matching actual safety.
  • Human Factors as a Foundation: The authors strongly suggest that the Human Factors discipline offers valuable existing knowledge and methodologies for understanding and managing trust in technical systems.
  • Anthropomorphism as a Tool: Drawing from previous work, the paper hints that anthropomorphizing automated systems (giving them human-like characteristics) might help reduce differences in trust and potentially manage perceived safety.
  • Requirements Engineering Needs a Human-Centric Shift: The core argument is that requirements engineering for safety-critical systems must explicitly address the human element of trust, moving beyond purely technical specifications.
  • Interdisciplinary Collaboration is Crucial: The paper highlights the necessity of bridging the gap between computer science/engineering and social sciences to effectively tackle the trust problem.

Key Results

This paper is primarily a conceptual and position paper, not one reporting empirical results. Therefore, there are no quantitative results to summarize. The "key results" are the insights and the proposed direction for future research and practice.

Limitations and Future Work

The authors explicitly acknowledge several limitations and propose future work:

  • Bridging Methodologies: A significant challenge identified is how to effectively map research questions and methods between Requirements Engineering and Human Factors, and vice versa.
  • Engineering Social Science Findings: A critical future direction is figuring out how to translate findings from human factors research into concrete engineering solutions that build trusted systems.
  • Computational Social Science: The paper suggests exploring the application of computational social science methods in both fields.
  • Technical Challenges: The authors pose several technical research questions, including:
  • Relating trust/undertrust/overtrust to non-functional requirements.
  • Developing evaluation techniques for actual vs. perceived safety.
  • Reducing trust manipulations via anthropomorphism.
  • Defining trust in decentralized blockchain systems.
  • Ensuring ongoing trust in evolving systems with machine learning.
  • Emerging Application Areas: The authors believe these challenges will become even more pressing and best tackled in the context of AI, autonomous adaptive systems, and blockchain.

Summary updated

BibTeX
@inproceedings{kaindl2019avoiding,
  author = {Hermann Kaindl and Davor Svetinovic},
  title = {{Avoiding Undertrust and Overtrust}},
  booktitle = {REFSQ Workshops},
  year = {2019},
  url = {https://openreview.net/forum?id=61ot20sFhR},
}
Figure