Participatory Design for Machine Learning Systems

87 concepts66 papers1 proposal

When machine learning models fail in the real world, the breakdown rarely stems from a lack of mathematical optimization. A model can achieve stellar test-set accuracy or satisfy formal engineering definitions of fairness, yet still amplify systemic inequities, disrupt clinical workflows, or fail to gain user trust. This friction occurs because traditional AI development operates in a top-down technical vacuum, where objectives are defined by a narrow group of engineers and corporate leaders (Blili-Hamelin et al.).

To bridge this gap, a growing paradigm shifts focus from post-hoc technical auditing to lifecycle-wide participatory design. By integrating diverse, non-technical stakeholders—including domain experts, ethicists, frontline workers, and impacted populations—directly into the development process, this approach treats AI systems not as isolated mathematical artifacts, but as products of collaborative socio-technical design.

The Failure Mode of Post-Hoc Auditing

The conventional approach to responsible AI relies heavily on technical auditing: developers train a model, evaluate it against a set of fairness or safety metrics, and attempt to apply algorithmic "fixes" if disparities emerge. However, Chen et al. argue that these technical patches are fundamentally insufficient because they fail to incorporate the lived experiences and domain-specific contexts necessary to detect deep-seated systemic biases.

When models are built without upstream stakeholder input, they risk optimizing for the wrong objectives entirely. For instance, Wolfe et al. demonstrated that AI models often absorb sensationalist media narratives about vulnerable populations, creating a total disconnect between algorithmic representations and lived realities. Similarly, in clinical settings, Schmidberger notes that prioritizing raw technical performance over patient-centered design leads to tools that are poorly optimized for actual care.

To prevent these misalignments, fairness and operational goals must be defined before the model is trained. Johnson et al. emphasize that integrating patients and clinicians during the initial design phase allows teams to establish equity goals that reflect real-world needs rather than abstract engineering metrics.

Operationalizing Values into Technical Requirements

A central challenge of participatory AI is translation: how do we convert qualitative human values or democratic deliberations into concrete, testable technical requirements? Holstein et al. point out that scaling these methods remains a significant hurdle due to the persistent difficulty of mapping user-defined needs to sophisticated data-driven models.

To solve this, researchers have adapted software engineering frameworks to the machine learning lifecycle. For example, Fung et al. introduced Accountability Driven Development (ADD), a framework that utilizes natural language "user stories" to define accountability, fairness, and explainability requirements before a single line of code is written. This allows non-technical stakeholders—such as ethicists and domain experts—to co-create testable scenarios, shifting the responsibility of system behavior from isolated engineers to a collaborative group.

Traditional Lifecycle:
[Data Collection] -> [Model Training] -> [Post-Hoc Technical Audit] -> [Deployment Failure]

Participatory Lifecycle:
[Multi-Stakeholder Deliberation] -> [Natural Language User Stories (ADD)] -> [Co-Designed Data/Objectives] -> [Model Training] -> [Context-Aware Deployment]

By formalizing stakeholder input early, developers can leverage social choice theory and participatory governance to ensure that the values encoded within a model reflect a legitimate aggregation of human intent, rather than arbitrary engineering defaults (Dai).

Scaffolding the Expertise Gap

A common objection to participatory design is that non-technical stakeholders lack the specialized knowledge required to contribute meaningfully to complex AI architectures. However, empirical evidence shows that when provided with appropriate communication scaffolding, non-experts offer highly specific, invaluable feedback.

Kuo et al. demonstrated this by using adapted "comicboarding" methods to elicit feedback from frontline workers and unhoused individuals regarding AI systems used in homeless services. Their findings show that individuals without formal AI expertise can provide precise insights into system deployment and design, directly challenging the assumption that AI governance must be restricted to technical experts.

When properly integrated, these stakeholders transition from passive users into active co-creators (Cai et al.; Jaiswal et al.). This collaborative loop expands the traditional user-AI dyad into a multi-stakeholder ecosystem. In healthcare, for instance, this means designing interfaces that facilitate communication across entire care teams—including patients, providers, and caregivers—rather than optimizing for an isolated user (Wang et al.; Spitale et al.).

Avoiding "Participation Washing"

As participatory methods gain traction—and become increasingly relevant for regulatory compliance under frameworks like the EU AI Act (Ullstein et al.)—the field faces the risk of "participation washing." This occurs when stakeholder engagement is treated as a superficial, tokenistic performance or a compliance checkbox rather than a meaningful driver of technical outcomes (Asghari et al.).

To combat this, the research community advocates for a fundamental shift in practice:

  • Co-creation over Consultation: Moving away from passive feedback loops toward authentic partnerships where communities hold active decision-making power (Hendricks‐Sturrup et al.). This is encapsulated by the principle of "Nothing for Us Without Us" in community-led initiatives (Pinhanez et al.).
  • Researching With, Not On: Shifting the paradigm when developing technologies for marginalized or indigenous groups by directly embedding them into the research and design process (NLP Progress in Indigenous Latin American Languages).
  • Open Practice over Open Access: Recognizing that true democratization requires more than just releasing open-source weights or code; it requires establishing structured, inclusive pathways for global communities to actively participate in building and governing models (Ding et al.).

Ultimately, participatory design works because it grounds machine learning systems in the messy, relational realities of human environments. By treating stakeholder integration as a core technical requirement rather than an administrative afterthought, the AI community can build systems that are not only accurate, but demonstrably trustworthy, equitable, and aligned with human well-being (Kunze; Goisauf et al.; Sun).

Key Foundational Work

Towards a New Participatory Approach for Designing Artificial Intelligence and Data-Driven Technologies (2021) — contributed 3 concept elements to this direction.
ExTRA: Explainable Therapy-Related Annotations (2020) — contributed 2 concept elements to this direction.
Build Back Better with Responsible AI (2021) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_participatory_ai_development_0aa42125,
  title = {participatory AI development},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/participatory-ai-development/rdr_0aa421257733428a996f7d1220166be5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_participatory_ai_development_0aa42125_v1,
  title = {participatory AI development},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/participatory-ai-development/rdr_0aa421257733428a996f7d1220166be5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake