Recentering Validity Considerations through Early-Stage Deliberations Around AI and Policy Design

Anna Kawakami, Amanda Coston, Haiyi Zhu, Hoda Heidari, Kenneth HolsteinarXiv 2023 · OpenReview

Abstract

AI-based decision-making tools are rapidly spreading across a range of real-world, complex domains like healthcare, criminal justice, and child welfare. A growing body of research has called for increased scrutiny around the validity of AI system designs. However, in real-world settings, it is often not possible to fully address questions around the validity of an AI tool without also considering the design of associated organizational and public policies. Yet, considerations around how an AI tool may interface with policy are often only discussed retrospectively, after the tool is designed or deployed. In this short position paper, we discuss opportunities to promote multi-stakeholder deliberations around the design of AI-based technologies and associated policies, at the earliest stages of a new project.

Introduction: The Validity Gap in AI Design

As AI-based decision-making tools are increasingly integrated into high-stakes domains such as child welfare, criminal justice, and healthcare, the research community has focused heavily on optimizing downstream properties like fairness, interpretability, and accuracy. However, this "technology-first" approach often overlooks a fundamental concern: validity.

A "validity gap" occurs when a tool is mathematically sound but practically misaligned with real-world goals. For instance, an AI tool in child welfare might be designed to predict "long-term placement out of home" as a proxy for risk. However, the human caseworker using the tool is legally and professionally mandated to assess "immediate safety risks." Because the design of the technology and the design of the governing policy are often treated as separate processes—scattered across different timelines and stakeholders—these misalignments are frequently only discovered after deployment, leading to systemic friction and potential harm.

A Framework for Recentering Validity

This paper proposes a conceptual shift away from retrospective policy-making toward a framework of Early-Stage Deliberation. This approach advocates for three core pillars:

  1. Early-Stage Deliberation: Conversations regarding law, ethics, and organizational objectives must occur at the very beginning of the development lifecycle, rather than as an afterthought.
  2. Multi-Stakeholder Engagement: The design of an AI’s objective function should formally involve legal experts, policymakers, frontline workers, and the community members most affected by the system.
  3. Integrated Co-Design: AI tools and the policies that govern them should be treated as a single, integrated system. One cannot determine if a tool is "valid" without understanding the specific legal and jurisdictional context in which it operates.

Key Insights and Challenges

The Proxy Problem

Real-world AI models almost always rely on imperfect proxies. In healthcare, a model might use "future healthcare costs" to represent "medical need." Without early policy deliberation, such a model may inadvertently prioritize wealthy patients over those with greater clinical needs, simply because the proxy fails to capture the intended construct of "need."

The Retrospective Trap

Currently, policy is often reactive. When policy is discussed only after a tool is deployed, it becomes nearly impossible to rectify fundamental flaws in the tool’s construct validity—the degree to which it actually measures what it claims to measure.

Power Imbalances

Traditional design processes often exclude those with the most at stake. Deliberative design must be intentionally structured to shift power toward frontline workers and community members, ensuring their expertise informs the tool's foundational logic.

Open Questions for Future Research

To move this framework into practice, the authors identify five critical questions for the Human-AI Interaction (HAI) and Science and Technology Studies (STS) communities:

  1. Shifting Power: How can shared language be developed to ensure community members have meaningful influence over technical design?
  2. Balancing Discourse: How can we maintain a balance between generative innovation and evaluative scrutiny without one stifling the other?
  3. Organizational Integration: How do we ensure that policies conceptualized during design sessions are actually adopted by local government bodies?
  4. Incentive Alignment: How can we combat the "move fast and break things" culture that discourages slow, deliberative processes?
  5. Alternative Solutions: Is deliberation always the optimal path, or are there other methods to ensure validity in high-stakes AI?

Conclusion

While early-stage deliberation is not a "silver bullet," it is a necessary step toward creating AI systems that are contextually valid and socially responsible. The authors acknowledge that stakeholder agreement does not guarantee successful implementation or performance. Future work must focus on operationalizing these theoretical concepts into practical, scalable tools and workshops that practitioners can use to bridge the gap between AI technology and public policy.

Summary updated

Abstract

AI-based decision-making tools are rapidly spreading across a range of real-world, complex domains like healthcare, criminal justice, and child welfare. A growing body of research has called for increased scrutiny around the validity of AI system designs. However, in real-world settings, it is often not possible to fully address questions around the validity of an AI tool without also considering the design of associated organizational and public policies. Yet, considerations around how an AI tool may interface with policy are often only discussed retrospectively, after the tool is designed or deployed. In this short position paper, we discuss opportunities to promote multi-stakeholder deliberations around the design of AI-based technologies and associated policies, at the earliest stages of a new project.
BibTeX
@misc{kawakami2023recentering,
  author = {Anna Kawakami and Amanda Coston and Haiyi Zhu and Hoda Heidari and Kenneth Holstein},
  title = {{Recentering Validity Considerations through Early-Stage Deliberations Around AI and Policy Design}},
  year = {2023},
  eprint = {2303.14602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.14602},
}
Figure