Despite the rapid adoption of artificial intelligence, current systems face significant hurdles regarding robustness, interpretability, and user trust. Historically, the AI community has relied on crowd computing primarily for "per-datum" labeling—asking workers simple, closed-ended questions like, "Is this a picture of a cat?"
This vision paper argues that this narrow approach is no longer sufficient. To address the systemic "brittleness" of modern AI, crowd computing must evolve into Crowd Conceptual Computing. By leveraging human intelligence at a higher conceptual level, researchers can build models that are not only more robust but also foster natural, trust-based interactions.
The Dual Challenges of Current AI
The authors identify two primary categories of failure in modern AI systems:
- Computational Challenges: Machine learning models are often "shallow." They tend to learn spurious correlations and fail predictably when encountering data even slightly outside their training distribution (the "unknown unknowns").
- Interactional Challenges: AI behavior is frequently unpredictable or inconsistent, violating standard usability guidelines. This lack of reliability erodes user confidence and often leads to the abandonment of the system.
The Paradigm Shift: Crowd Conceptual Computing
Rather than presenting a new algorithm, this paper proposes a foundational shift in how we utilize human-in-the-loop systems. The core components of Crowd Conceptual Computing include:
- Knowledge-Level Contribution: Shifting crowd work from labeling individual data points to contributing high-level conceptual knowledge. This approach aims to bridge the gap between the flexibility of deep learning and the logic and causality of symbolic AI.
- Neurosymbolic Integration: Utilizing crowds to provide the "symbolic" knowledge—concepts and relationships—needed to make deep learning models more explainable and robust.
- Open-Ended Task Design: Transitioning away from rigid tasks toward complex, collaborative, and creative knowledge creation.
- Conversational Interaction: Adopting conversational interfaces (such as chatbots or voice assistants) as the primary medium for both crowd workers and end-users. This minimalist, human-like interaction style can increase worker engagement and user trust.
Key Insights for Future AI Systems
By elevating the role of the crowd, developers can address several critical bottlenecks in AI development:
- Solving the "Unknown Unknowns": Models fail when they lack specific concepts. Instead of blindly adding more training data, crowds can be deployed to identify exactly which concepts are missing or underrepresented in a model's "worldview."
- Matching Human Mental Models: Current interpretability methods, such as heatmaps showing which pixels a model focused on, fail to explain why a decision was made in human terms. Crowds can help align model explanations with actual human reasoning.
- Closing the Semantic Gap: There is often a disconnect between local interpretability (why a specific instance was classified) and global interpretability (how the model works overall). Conceptual crowd computing allows humans to query models using multi-concept logic, bridging this gap.
Roadmaps and Future Challenges
Ultimately, this paper provides a roadmap for the next generation of Human-in-the-loop AI (focusing on robustness and interpretability) and Human-AI Interaction (focusing on decision support and congruence with human understanding).
However, realizing this vision requires overcoming several hurdles:
- The Complexity of Knowledge Creation: Generating conceptual knowledge for real-world tasks is an open-ended, difficult process that extends far beyond the capabilities of existing knowledge bases.
- Understanding Human Impact: The long-term impact of AI on human decision-making remains largely unexplored. Interdisciplinary research is needed to understand how different stakeholders perceive AI-generated explanations.
- Personalizing Explanations: An explanation that satisfies a data scientist might confuse a medical doctor or a layperson. Future systems must learn to adapt and personalize AI interactions across diverse user groups.
Summary updated