Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge

Anna Kawakami, Luke Guerdan, Yanghuidi Cheng, Kate Glazko, Matthew L. Lee, Scott A. Carter, Nikos Aréchiga, Haiyi Zhu, Kenneth Holstein

Introduction

In high-stakes domains such as child welfare, the integration of AI into human decision-making processes is often evaluated through "appropriate reliance"—the degree to which a human follows an AI's correct predictions and rejects its errors. However, this metric assumes a perfect alignment between the AI's training target and the human's ultimate goal. This paper, a collaboration between Carnegie Mellon University, Toyota Research Institute, and the University of Washington, introduces the concept of Critical Use. Critical use is defined as the human ability to weigh AI predictions against unique, qualitative knowledge, effectively situating the model's output within a broader context.

The Challenge: Mismatched Targets and Information Asymmetry

Standard AI-assisted decision-making faces two primary hurdles in real-world applications:

  1. Target-Construct Mismatch: AI models are often trained on "proxies" (e.g., predicting foster care placement within two years) while human experts prioritize "constructs" (e.g., ensuring immediate child safety).
  2. Information Asymmetry: Humans often have access to "unobservables"—qualitative data like the tone of a referral call or nuanced family dynamics—that are not captured in the structured data used to train the AI.

The authors argue that training should focus on helping humans understand when to exercise discretion rather than simply teaching them to "trust" the system.

Experimental Methodology

The researchers conducted a randomized online experiment with 354 participants, including crowd workers and social work students. The study utilized the Allegheny Family Screening Tool (AFST), a real-world risk assessment tool used in child maltreatment screening.

Experimental design flowchart showing the progression from onboarding and pre-assessment through training conditions to post-assessment and final questions.

Participants were guided through a structured workflow (see Figure 1). During the training phase, they were split into two conditions: "Practice Only" and "Practice + Explicit Feedback." The latter group received historical outcomes and information on how experienced social workers handled the same cases.

The case overview interface provided to participants, featuring a qualitative case description and structured overview of the referral.

As shown in Figure 2, the interface provided participants with both structured data and qualitative narratives. This allowed the study to measure how participants integrated "unobservable" information from the case descriptions with the AI's numerical risk score.

Key Findings

The study revealed several counter-intuitive insights regarding how humans learn to interact with AI:

  • Learning to Disagree: Practice did not lead to higher reliance on the AI. Instead, "Model Agreement" dropped from over 70% to 42% by the end of the study. Participants learned to stop uncritically accepting the AI's "nudge" as they gained experience with the cases.
  • Alignment with Expertise: While agreement with the AI decreased, "Worker-based Accuracy"—alignment with experienced social workers—increased from 52% to 61%. This suggests that participants were developing professional-like discretion.
  • The Role of Narrative: Post-test analysis showed that participants mentioned the AI score 73% less often in their justifications, shifting their focus toward immediate safety risks identified in the text narratives.
  • Feedback as a Distraction: Surprisingly, explicit historical feedback did not outperform simple practice. Those in the "Practice Only" group actually became better at predicting the AI's behavior, suggesting that "right/wrong" labels might interfere with learning the model's underlying logic.

Conclusion

This research highlights a paradigm shift for AI development. The goal of human-AI collaboration should not be to maximize reliance, but to empower humans to be the AI's most informed critics. By providing opportunities for situated practice, even novices can learn to identify the limitations of algorithmic proxies and prioritize the complex, qualitative realities of high-stakes decision-making.

Summary updated

BibTeX
@inproceedings{kawakami2023training,
  author = {Anna Kawakami and Luke Guerdan and Yanghuidi Cheng and Kate Glazko and Matthew L. Lee and Scott A. Carter and Nikos Aréchiga and Haiyi Zhu and Kenneth Holstein},
  title = {{Training Towards Critical Use: Learning to Situate AI Predictions Relative to Human Knowledge}},
  booktitle = {CI},
  year = {2023},
  doi = {10.1145/3582269.3615595},
}

Loading related content...

Figure