Human involvement in autonomous decision-making systems. Lessons learned from three case studies in aviation, social care and road vehicles

Pericle Salvini, Tyler Reinmund, Benjamin Hardin, Keri Grieman, Carolyn Ten Holter, Aaron Johnson, Lars Kunze, Alan Winfield, Marina JirotkaFrontiers in Political Science 2023 · DOI

Abstract

This paper draws on three case studies to examine the challenges and tensions involved in the use of Autonomous Decision-Making Systems (ADMS). In particular, the paper highlights: (i) challenges around the shifting “locale” of the decision and associated consequences for stakeholders; (ii) potential implications for stakeholders from regulation such as the General Data Protection Regulation (GDPR); (iii) the different values that stakeholder groups bring to the “decision” question; (iv) how complex pre-existing webs of stakeholders and decision-making authorities may be disrupted or disempowered by the use of an automated system and the lack of evaluation of possible consequences; and (v) how ADMS for non-technical users can lead to circumvention of the boundaries of intended system use. We illustrate these challenges through case studies in adult social care, aviation, and vehicle driver monitoring systems, concluding with recommendations for practice and policy.

Introduction: The Paradox of Control

As we increasingly delegate high-stakes choices to AI—from medical triage to flight stabilization—we face a fundamental tension known as the "paradox of control." While ADMS are deployed to reduce human error and operational costs, society demands "meaningful human control" to ensure accountability and ethics.

Current regulatory frameworks, such as the UK GDPR’s Article 22 (which grants the right not to be subject to "solely" automated decisions), are often too vague for real-world engineering and policy. This paper tackles the challenge of defining where a decision actually "lives" in a complex system and how human oversight can easily degrade into "rubber-stamping" (quasi-automation) or be circumvented by users entirely.

Methodology and Data Sources

The researchers employed a multi-domain case study approach to identify cross-cutting challenges in ADMS deployment:

  • Adult Social Care: A five-month participant-observation study within a UK local authority implementing an ADMS to predict fall risks in the elderly. This involved analyzing unstructured practitioner case notes processed via Natural Language Processing (NLP).
  • Aviation: A forensic analysis of the Boeing 737 MAX MCAS failures, utilizing publicly available investigative reports and flight data recorder (black box) analysis from the Lion Air 610 and Ethiopian 302 crashes.
  • Automotive Safety: A literature review of Driver Monitoring Systems (DMS), examining existing research datasets on driver gaze behavior, physiological responses (ECG/GSR), and steering wheel torque measurements.

The findings were synthesized and mapped against the European Law Institute (ELI) Guiding Principles to offer a structured reflection on how policy meets practice.

Key Insights: Shifting Locales and System Authority

The Shifting "Locale" of Decisions

Decisions are rarely a single point in time. In the social care case, the "decision" could be the initial risk score, the act of sharing that data, or the final care allocation. If oversight only happens at the end of the pipeline, the "upstream" biases are already baked in, making human intervention performative rather than substantive.

The False Negative Blind Spot

Regulation like the GDPR focuses heavily on people wrongly included by an algorithm (false positives). However, the authors found a massive ethical gap regarding false negatives—those the system misses who are then denied preventative care without any human ever reviewing their case. In resource-constrained environments, these individuals effectively disappear from the decision-making process.

Automation as a "Patch"

The Boeing MCAS case illustrates the danger of using software to "fix" aerodynamic instability caused by hardware changes. When ADMS is a "bolt-on" fix rather than an added value, it can lead to catastrophic failures if the human operator isn't fully aware of the system's existence, authority, or failure modes.

The Incentive to Circumvent

In semi-autonomous vehicles, the "human-in-the-loop" often becomes a "human-as-monitor," which is cognitively exhausting. This creates an incentive for users to "game" the system (e.g., using weights to trick steering wheel sensors), effectively bypassing the safety measures intended to protect them. This highlights a disconnect between intended system use and actual consumer behavior.

Key Results and Feasibility

  • Oversight Feasibility: In the social care study, manual human review of every algorithmic output was deemed "disproportionate" and unfeasible for a population-level program, leading to a reliance on "quasi-automation."
  • System Authority: The Boeing case revealed that MCAS was given excessive authority (tilting the stabilizer by 2.5 degrees vs. the original 0.6) while relying on a single, non-redundant sensor—a violation of basic safety engineering principles.
  • Detection Speed: In automotive safety, camera-based DMS were found to detect driver distraction 50 seconds sooner than traditional steering-wheel-based torque sensors, yet the latter remain common due to lower costs and fewer privacy hurdles.

Conclusion and Policy Recommendations

The findings suggest that current approaches to human involvement are often insufficient. The authors propose several paths forward:

  1. Ethical Risk Assessments (ERA): Moving beyond simple safety checks to systematically identify societal and ethical harms before deployment.
  2. Standardized Transparency: Implementing standards like IEEE 7001 to ensure transparency is tailored to different stakeholders (e.g., what a pilot needs to know vs. what a regulator needs).
  3. Independent Evaluation: Shifting away from self-certification by developers toward mandatory, independent expert audits for high-stakes ADMS.

While this study is limited to three domains, it underscores the need for a more nuanced understanding of how automated systems disrupt existing webs of authority and the critical importance of evaluating consequences before wide-scale deployment.

Summary updated

Abstract

This paper draws on three case studies to examine some of the challenges and tensions involved in the use of Autonomous Decision-Making Systems (ADMS). In particular, the paper highlights: (i) challenges around the shifting “locale” of the decision, and the associated consequences for stakeholders; (ii) potential implications for stakeholders from regulation such as the General Data Protection Regulation (GDPR); (iii) the different values that stakeholder groups bring to the “decision” question; (iv) how complex pre-existing webs of stakeholders and decision-making authorities may be disrupted or disempowered by the use of an automated system and the lack of evaluation of possible consequences; (v) how ADMS for non-technical users can lead to circumvention of the boundaries of intended system use. We illustrate these challenges through case studies in three domains: adult social care, aviation, and vehicle driver monitoring systems. The paper closes with recommendations for both practice and policy in the deployment of ADMS.
BibTeX
@inproceedings{salvini2023human,
  author = {Pericle Salvini and Tyler Reinmund and Benjamin Hardin and Keri Grieman and Carolyn Ten Holter and Aaron Johnson and Lars Kunze and Alan Winfield and Marina Jirotka},
  title = {{Human involvement in autonomous decision-making systems. Lessons learned from three case studies in aviation, social care and road vehicles}},
  booktitle = {Frontiers in Political Science},
  year = {2023},
  doi = {10.3389/fpos.2023.1238461},
}
Figure