BiO-HMC: Dynamic Human-Machine Collaboration for Consensus Decision-Making via Bilevel Optimization

Yinghui Pan, Shuaijie Zhao, Shenbao Yu, Zongyang Liu, Yifeng Zeng, Han Liu, Mingwei LinAAAI 2026 · DOI

Abstract

Consensus decision-making uses crowd responses (usually from non-experts) to questions to reach a consensus answer based on human-machine collaboration. The crucial point is dynamic, which should not only enable rapid self-iteration toward the correct answer through crowd workers' responses but also adaptively suggest the next most valuable question(s) to accelerate the integration of the answer. However, existing methods reach consensus using either offline data or fixed question search structures, thereby largely sidestepping this dynamic nature. In response, we propose a bilevel optimization-based human-machine collaboration (BiO-HMC), which explores an inner & outer-level optimization to enable effective answer integration and efficient question selection. The resulting optimization problem is intractable because there is no closed-form expression in the inner-level optimization. We employ a gradient-based method and guarantee the method's theoretical convergence. Experimental results on synthetic and real-world datasets demonstrate the effectiveness and efficiency of the BiO-HMC model, i.e., achieving the highest confidence in the correct answer with the lowest labor cost.

Introduction

In crowdsourcing, we often rely on non-experts to solve complex tasks, such as classifying a galaxy or identifying a rare plant species. Because these workers aren't specialists, we don't ask them for the final answer directly. Instead, we ask a series of simpler, feature-based questions (e.g., "Does this galaxy have a spiral shape?").

The challenge lies in two areas: answer integration (how to combine noisy, sometimes conflicting responses into a single "truth") and question selection (how to decide which question to ask next to reach that truth efficiently). Existing methods often use "static" approaches—relying on offline data or fixed decision trees—which fail to adapt to the specific, real-time responses provided by workers. This leads to wasted effort, higher costs, and lower accuracy when the task gets complex.

Methodology: The BiO-HMC Framework

The authors propose BiO-HMC (Bilevel Optimization-based Human-Machine Collaboration), which treats consensus-building as a nested optimization problem. This approach allows the machine to act as a "smart interviewer," actively choosing the next question based on previous answers rather than acting as a passive aggregator.

Overview of the BiO-HMC framework showing the interaction between worker sequences and machine summarization.

1. The Outer-Level Optimization (Belief Updating)

This layer uses a specialized Bayesian Network called the $NO_2$ model (Noisy-OR). It treats the candidate answers as "latent" (hidden) variables and the workers' responses as observed variables. As new responses come in, the system updates its "belief" (probability) for each possible answer. The Noisy-OR structure accounts for the fact that a single feature (like "smoothness") might support multiple different categories.

2. The Inner-Level Optimization (Question Selection)

Instead of asking questions in a random or fixed order, this layer identifies the "most valuable" question. It uses a pairwise value preference matrix to rank questions based on how much they are expected to clarify the current uncertainty.

3. Gradient-Based Solver

Because the inner loop doesn't have a simple mathematical solution (it's "intractable"), the authors employ a stochastic gradient-based method to solve the optimization. They also provide a theoretical proof that this method will eventually converge to an optimal state.

Experimental Evaluation

The model was evaluated on two types of data:

  • Synthetic Datasets (Sync1-5): Five generated datasets with increasing levels of complexity in the relationships between questions and answers, used to test the model under controlled worker error levels.
  • Galaxy Zoo (GZ): A real-world dataset containing 45 candidate galaxy categories and over 17,000 worker responses. This served as a high-stakes test for the model's performance in a complex scenario.
Performance comparison (HSR) of BiO-HMC against baselines across synthetic and real-world datasets.

Key Results

  • Superior Accuracy: BiO-HMC consistently achieved the Highest Support Rate (HSR)—often exceeding 98%—outperforming state-of-the-art baselines like MC-VOI and PMC-Cnts.
  • Drastic Cost Reduction: In the Galaxy Zoo task, BiO-HMC reached a consensus using only 287 interactions, whereas some baseline methods required over 9,000 or even 25,000 interactions to reach similar confidence levels.
  • Robustness to Noise: Even when simulated workers were wrong 50% of the time, BiO-HMC was able to recover and identify the correct answer as more workers were added, while baseline methods often plateaued.

Conclusion and Future Work

BiO-HMC demonstrates that efficiency in crowdsourcing comes from asking the right questions to the right workers. While highly effective, future work could improve the model by incorporating individual worker skill levels and extending the system to detect "unknown" answers not present in the initial training data. Additionally, optimizing the computational complexity of the real-time gradient calculations remains an area for further refinement in extremely large-scale applications.

Summary updated

Abstract

Consensus decision-making uses crowd responses (usually from non-experts) to questions to reach a consensus answer based on human-machine collaboration. The crucial point is dynamic, which should not only enable rapid self-iteration toward the correct answer through crowd workers' responses but also adaptively suggest the next most valuable question(s) to accelerate the integration of the answer. However, existing methods reach consensus using either offline data or fixed question search structures, thereby largely sidestepping this dynamic nature. In response, we propose a bilevel optimization-based human-machine collaboration (BiO-HMC), which explores an inner & outer-level optimization to enable effective answer integration and efficient question selection. The resulting optimization problem is intractable because there is no closed-form expression in the inner-level optimization. We employ a gradient-based method and guarantee the method's theoretical convergence. Experimental results on synthetic and real-world datasets demonstrate the effectiveness and efficiency of the BiO-HMC model, i.e., achieving the highest confidence in the correct answer with the lowest labor cost.
BibTeX
@inproceedings{pan2026bio,
  author = {Yinghui Pan and Shuaijie Zhao and Shenbao Yu and Zongyang Liu and Yifeng Zeng and Han Liu and Mingwei Lin},
  title = {{BiO-HMC: Dynamic Human-Machine Collaboration for Consensus Decision-Making via Bilevel Optimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i21.38820},
}
Figure