Causal Learning Should Embrace the Wisdom of the Crowd

Ryan Feng Lin, Yuantao Wei, Huiling Liao, Xiaoning Qian, Shuai HuangarXiv 2026 · DOI · arXiv

Abstract

Learning causal structures typically represented by directed acyclic graphs (DAGs) from observational data is notoriously challenging due to the combinatorial explosion of possible graphs and inherent ambiguities in observations. This paper argues that causal learning is now ready for the emergence of a new paradigm supported by rapidly advancing technologies, fulfilling the long-standing vision of leveraging human causal knowledge. This paradigm integrates scalable crowdsourcing platforms for data collection, interactive knowledge elicitation for expert opinion modeling, robust aggregation techniques for expert reconciliation, and large language model (LLM)-based simulation for augmenting AI-driven information acquisition. We focus on DAG learning for causal discovery and frame the problem as a distributed decision-making task, recognizing that each participant possesses fragmented and imperfect knowledge. By proposing a systematic framework to synthesize these insights, we aim to enable the recovery of a global causal structure unachievable by any individual agent alone.

Introduction

Inferring causal relationships—represented as Directed Acyclic Graphs (DAGs)—is a cornerstone of science, yet it remains a computationally "notoriously challenging" problem. Purely data-driven methods struggle with computational intractability, as the search space for possible graphs grows super-exponentially with the number of variables. Furthermore, statistical ambiguity often prevents observational data from distinguishing between different causal directions, leaving researchers with a Markov equivalence class rather than a single true graph.

While researchers often turn to domain experts to resolve these ambiguities, traditional approaches rely on a single expert or a small group. This is fragile: experts have fragmented and imperfect knowledge, biases, and limited bandwidth. This paper proposes a shift toward a decentralized "wisdom of the crowd" paradigm, combining noisy insights from human experts and LLM-based agents to reconstruct complex causal structures.

Methods

The authors frame causal discovery as a distributed decision-making task. Their framework consists of several key pillars:

  • Knowledge Elicitation Frameworks: Comparing edge-wise queries (direct links) versus ordering-wise queries (relative position in causal flow).
  • Aggregation Strategies: Moving from expert-level aggregation (combining full graphs) to query-level aggregation, which better preserves uncertainty and "puzzle-piece" knowledge.
  • Optimal Design of Elicitation: Using active learning to prioritize queries with the highest Expected Information Gain.
  • Agent-based Expert Simulation: Utilizing Large Language Models (LLMs) as surrogate agents to simulate expert reasoning and bootstrap acquisition.

Datasets and Taxonomy

The primary testbed for this framework is the classic "Asia" network, an 8-node Bayesian network representing respiratory disease diagnosis.

The Asia network testbed, illustrating different domains of causal knowledge such as travel history and smoking-related diseases.

To model the diverse quality of crowd contributions, the authors define a taxonomy of expert types based on completeness, belief validity, confidence, and trustworthiness.

Taxonomy of expert types used to model fragmented and imperfect knowledge in causal discovery.

Key Insights

  • Ordering > Edges: Ordering-wise knowledge is often more query-efficient and robust than edge-wise knowledge, providing soft structural constraints that are easier to denoise.
  • LLMs as Knowledge Amplifiers: LLMs can act as simulators to understand the "causal world in an expert’s mind," generating plausible structures with minimal human intervention.
  • Crowd-in-the-loop: Reliability emerges not from a single "vetted" expert, but from the post-hoc processing of a diverse, decentralized crowd.

Key Results

A pilot study with 20 human participants confirmed that expert causal knowledge is far from uniform. Participant scores showed massive variance in confidence and accuracy across different queries. Some causal links reached broad consensus, while others induced significant disagreement, highlighting the high-dimensional nature of expert knowledge. The framework successfully demonstrated that by modeling these variations, it could identify "Bad Actors" or "Uncertain" experts and weigh their contributions to recover an accurate global structure.

Limitations and Future Work

Future research must address incentives for high-quality participation and protection against sophisticated adversarial behavior. Additionally, there is a need for formal information-theoretic lower bounds on query requirements and a clearer understanding of the optimal division of labor between human experts and LLM agents.

Summary updated

Abstract

Learning causal structures typically represented by directed acyclic graphs (DAGs) from observational data is notoriously challenging due to the combinatorial explosion of possible graphs and inherent ambiguities in observations. This paper argues that causal learning is now ready for the emergence of a new paradigm supported by rapidly advancing technologies, fulfilling the long-standing vision of leveraging human causal knowledge. This paradigm integrates scalable crowdsourcing platforms for data collection, interactive knowledge elicitation for expert opinion modeling, robust aggregation techniques for expert reconciliation, and large language model (LLM)-based simulation for augmenting AI-driven information acquisition. In this paper, we focus on DAG learning for causal discovery and frame the problem as a distributed decision-making task, recognizing that each participant (human expert or LLM agent) possesses fragmented and imperfect knowledge about different subsets of the variables of interest in the causal graph. By proposing a systematic framework to synthesize these insights, we aim to enable the recovery of a global causal structure unachievable by any individual agent alone. We advocate for a new research frontier and outline a comprehensive framework for new research thrusts that range from eliciting, modeling, aggregating, and optimizing human causal knowledge contributions.
BibTeX
@misc{lin2026causal,
  author = {Ryan Feng Lin and Yuantao Wei and Huiling Liao and Xiaoning Qian and Shuai Huang},
  title = {{Causal Learning Should Embrace the Wisdom of the Crowd}},
  year = {2026},
  eprint = {2603.02678},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02678},
}
Figure