Principled Human-AI Collaboration in Agentic Workflows

Young Hyun Cho, Will Wei SunarXiv 2026 · arXiv

TL;DR

This paper introduces a statistical framework for safely delegating tasks to AI agents within iterative "generate–verify" workflows. By using a limited number of human expert evaluations to calibrate black-box systems, the authors provide a way to maximize automation while guaranteeing that released outputs meet strict quality standards with a pre-defined level of confidence.

What Problem Does This Paper Address?

As organizations scale "agentic" AI—systems that can reason, use tools, and self-correct—they face a fundamental tension: how do you reap the productivity gains of automation without letting low-quality or hallucinated outputs slip through?

Current workflows often use a generate–verify loop, where one model proposes an answer and another (the verifier) scores it. If the score is low, the generator tries again. However, verifier scores are imperfect proxies for human-level quality. A high score doesn't guarantee a correct answer, especially in high-stakes domains like finance or medicine.

The authors address the challenge of safe delegation: determining exactly which verifier thresholds and attempt limits allow an organization to release AI results without human review, while ensuring the error rate stays below a specific tolerance (e.g., <5%).

Figure 1: Generate–verify workflow, adapted from the workflow in Anthropic (2024). A generator proposes a candidate, and a verifier assigns a score and provides feedback. The generator may revise the answer in another round if it does not pass verification.

Methods

The researchers developed a statistical calibration method that treats the AI generator and verifier as "black boxes"—meaning you don't need to retrain the models or even know how they work internally.

  1. Pilot Phase: Before deployment, the system runs on a set of pilot tasks. Human experts evaluate a subset of these outputs, providing binary "correct/incorrect" labels based on their own professional standards.
  2. Simultaneous Hypothesis Testing: The framework treats every possible combination of verifier thresholds ($s$) and round limits ($m$) as a candidate "deployment rule." It uses the weighted Holm procedure to test these rules against the human labels. This ensures that the probability of accidentally approving a "bad" rule (one that exceeds the error tolerance) is strictly controlled.
  3. Optimization: Among all rules that pass the statistical safety check, the system selects the one that yields the highest estimated automation rate.
Overview of the framework: human experts calibrate the system pre-deployment to ensure safe autonomous operation during deployment.

Datasets

  • MultiHiertt: A complex financial question-answering benchmark based on real annual reports. It requires reasoning over hierarchical tables and text, making it a "stress test" for AI agents.
  • Synthetic Data: Used to rigorously test the theoretical limits of the framework, specifically how the "quality margin" (how much better the AI is than the required threshold) affects the number of human evaluations needed.

Key Insights and Ideas

  • The "Quality Margin" is King: The number of human evaluations required to certify a workflow depends heavily on the gap between the AI's actual performance and the organization's error tolerance. If an AI is barely good enough, you need significantly more human labels to prove it's safe.
  • Heuristics are Risky: Common industry heuristics—like "release the answer if the verifier gives it a high score three times in a row"—often fail. In the MultiHiertt study, these heuristics either violated safety bounds or were so conservative they automated almost nothing (<1.3%).
  • Fundamental Limits: The authors derived a minimax lower bound, proving that the relationship between required human effort and the quality margin ($1/\Delta_e^2$) is a fundamental statistical reality, not just a limitation of their specific algorithm.

Key Results

In the financial QA case study (MultiHiertt), the framework demonstrated impressive scaling:

  • Automation Gains: As the number of expert evaluations increased from 1,000 to 5,000, the average automation rate jumped from 2.50% to 24.17%.
  • Safety Guarantee: Despite the increase in automation, the error rate of released outputs remained rock-solid at approximately 1.20%, well below the 5% maximum allowed.
  • Efficiency: On challenging tasks, the system correctly identified that it couldn't meet strict 5% error targets with the available data and chose to escalate to humans, whereas it successfully automated more tasks when the tolerance was relaxed to 10%.
The probability of achieving target automation increases as more human evaluations are collected, especially when the quality margin is larger.

Limitations and Future Work

  • Representative Sampling: The method assumes the pilot tasks are representative of what the AI will see during deployment. If the data distribution shifts, the safety guarantees may weaken.
  • Expert Agreement: The framework relies on human experts as the "gold standard." The authors note that future work could explore how to handle disagreements between multiple human evaluators.
  • Prior Knowledge: Currently, the model starts "from scratch" for each task. Integrating prior information about an AI's general performance (e.g., from public benchmarks) could reduce the number of new human evaluations needed for a specific organizational task.

Summary updated

Abstract

Organizations adopting AI face the challenge of deciding which outputs can be released without human review while meeting the quality standards of human experts. We develop a statistical framework for delegating tasks between experts and black-box generate-verify workflows, in which AI-generated answers are verified and revised over multiple rounds. Before deployment, human experts evaluate selected outputs, and their judgments are used to calibrate the verifier threshold and number of attempts. Our method selects the policy with the highest estimated automation rate among statistically approved candidates, without modifying the generator or verifier. We establish finite-sample control of the error rate among released outputs, characterize the expert evaluations sufficient to attain a target automation rate, and derive a minimax lower bound establishing rate optimality in the quality margin and required confidence. In a MultiHiertt financial question-answering case study, average held-out automation increases from 2.50% to 24.17% as expert evaluations accumulate, while released-output error rates remain around 1.20%. In contrast, heuristics that release answers after consistently high verifier scores either violate the 5% error tolerance or automate at most 1.30% of tasks. Synthetic experiments further examine these theoretical predictions under controlled settings.
BibTeX
@misc{cho2026principled,
  author = {Young Hyun Cho and Will Wei Sun},
  title = {{Principled Human-AI Collaboration in Agentic Workflows}},
  year = {2026},
  eprint = {2611.01566},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01566},
}
Figure