Editorial: Human-centered AI: Crowd computing

Jie Yang, Alessandro Bozzon, Ujwal Gadiraju, Matthew LeaseFrontiers Artif. Intell 2023 · DOI

Introduction

The rapid advancement of Artificial Intelligence (AI) and Machine Learning (ML) over the last fifteen years has been largely driven by an "insatiable hunger" for human-labeled data. However, the field is currently undergoing a paradigm shift. As the industry moves beyond the initial explosion of scale, it is encountering significant hurdles where sheer volume no longer guarantees performance. This editorial explores a curated collection of research that advocates for a transition from "more data" to "better data" through human-centered design, emphasizing the critical role of human computation (HCOMP) and crowdsourcing in the development of sophisticated, hybrid human-AI systems.

From Scale to Data Excellence

The traditional approach to AI development often treats human labor as a commodity, focusing on the acquisition of millions of labels without sufficient regard for their quality or ethical context. The authors argue that this neglect leads to "perilous" failures in deployed systems. The emerging field of Data-Centric AI addresses these challenges by prioritizing "data excellence" over data scale. This involves ensuring high-quality annotations, managing "last-mile" capabilities where AI requires real-time human intervention, and addressing the ethical implications of "ghost work"—the often-exploited hidden human labor that powers modern AI.

Methodological Advances in Human Computation

To achieve data excellence, researchers are developing novel workflows and mathematical techniques to refine the interaction between humans and machines. Key methodologies highlighted in this collection include:

  • The FIND-RESOLVE-LABEL Workflow: Developed by Pradhan et al., this three-stage process reduces ambiguity in task instructions. By allowing workers to provide feedback and resolve uncertainties before the final labeling phase, the system significantly lowers worker disagreement rates.
  • Noise Estimation via Temperature Scaling: Uma et al. introduced a technique to mathematically estimate the level of noise and disagreement within crowdsourced datasets, providing a more rigorous way to evaluate data reliability.
  • Gamified Explainable AI (XAI): Tocchetti et al. utilized gamification to educate crowd workers about "black-box" models. This approach improves the crowd's ability to understand, interact with, and provide meaningful feedback to complex AI systems.
  • Tiered Crowdsourcing for Knowledge Graphs: Welty et al. implemented a three-tier approach to elicit class-level attributes, effectively solving the "label sparsity" problem where AI lacks sufficient examples to learn rare or niche categories.

The Evolving Role of the Human in AI Systems

A central insight of this research is the redefinition of the human role from simple "pre-processors" to active components of the system architecture. Humans are increasingly functioning as "Human Processing Units" (HPUs) or "Remote Person Calls" (RPCs). In this "Human API" model, AI systems call upon human intelligence at run-time to handle edge cases and complex scenarios that exceed current algorithmic capabilities.

Furthermore, studies on team formation (Vinella, Hu et al.) demonstrate that human agency often outperforms automated selection. In open collaboration environments like hackathons, teams formed by humans themselves were found to be more competitive and effective than those assembled by algorithms, highlighting the importance of social dynamics and self-organization.

Empirical Results and Socio-Technical Insights

The editorial synthesizes several key findings that challenge common assumptions about crowd work:

  • Worker Expertise: Research by Samiotis et al. in the domain of music perception found that untrained workers often possess high perception skills, making them valuable for specialized annotation tasks previously thought to require expert knowledge.
  • Confidence as a Feature: Yasmin et al. showed that incorporating self-reported worker confidence as a feature in classifiers leads to higher accuracy than using labels alone.
  • Personality and Performance: In high-stress, time-sensitive tasks such as incident response, specific personality traits and communication patterns were identified as the primary drivers of successful team performance.

Conclusion and Future Outlook

The future of human-centered AI lies at the intersection of AI, Human-Computer Interaction (HCI), and social sciences. To build truly robust and trustworthy systems, the field must move toward socio-technical solutions that prioritize:

  1. Responsible Sourcing: Transitioning away from exploitative labor models toward practices that provide fair economic opportunities.
  2. FAIR Principles: Ensuring that crowd science is Findable, Accessible, Interoperable, and Reproducible.
  3. Transparency and Bias Mitigation: Developing systems that are not only accurate but also fair and transparent, actively working to identify and mitigate the propagation of human biases through training data.

By shifting the focus toward data excellence and the "wisdom of the crowd," the research community can ensure that the next generation of AI systems is both safer and more effective.

Summary updated

BibTeX
@article{yang2023editorial,
  author = {Jie Yang and Alessandro Bozzon and Ujwal Gadiraju and Matthew Lease},
  title = {{Editorial: Human-centered AI: Crowd computing}},
  journal = {Frontiers Artif. Intell.},
  year = {2023},
  doi = {10.3389/frai.2023.1161006},
}
Figure