AutoAnalysis: An automated, self-updating meta-analysis of human-AI collaboration

Michelle Vaccaro, Abdullah Almaatouq, Thomas W. MalonearXiv 2026 · arXiv

TL;DR

AutoAnalysis is an LLM-powered pipeline that automates the systematic review process to keep pace with the exponential growth of human-AI collaboration research. Its most striking finding is a persistent "synergy gap": while AI consistently helps humans perform better than they would alone, these teams usually underperform compared to the best standalone agent—whether that’s the human or the AI.

What Problem Does This Paper Address?

As generative AI becomes a staple in the workplace, researchers are flooding the literature with studies on how humans and AI should work together. However, the results are often contradictory—some studies show massive productivity gains, while others highlight significant performance dips.

Traditionally, we rely on meta-analyses to reconcile these conflicting findings. The problem? Manual meta-analyses are incredibly labor-intensive and are often outdated by the time they are published. Given the current explosion of research in this field, a static review is no longer sufficient.

Figure 1: Published papers with “human-AI” in the title, 2000–2026.

Methods

The authors introduce AutoAnalysis, a "living" meta-analysis pipeline that uses frontier AI agents to automate every stage of a systematic review.

The pipeline follows a pre-registered protocol:

  1. Discovery: It starts with a "seed" of papers and uses a "snowballing" technique—automatically searching for forward and backward citations via the OpenAlex API.
  2. Screening & Extraction: Specialized LLM agents (specifically GPT-6.1-sol and GPT-6-astra) screen abstracts, read full-text PDFs, and extract quantitative data (means, SDs, sample sizes) and moderator variables (task type, AI explanations, etc.).
  3. Synthesis: The system employs a three-level random-effects model to calculate pooled effect sizes while accounting for the fact that multiple results often come from the same paper.
  4. Validation: To ensure accuracy, the authors implemented "agent audits" where a different model family (Claude Fable 5.1) re-evaluates a sample of the primary agents' decisions, alongside manual human sanity checks.
Figure 2: The AutoAnalysis pipeline.

Datasets

AutoAnalysis doesn't use a fixed training dataset; instead, it operates on a dynamic corpus of experimental literature. As of October 2026, the system had processed:

  • 15,495 unique records identified through snowballing.
  • 420 papers meeting strict inclusion criteria (original experiments with human participants and generative AI).
  • 1,955 unique effect sizes synthesized in the final model.

Key Insights and Ideas

  • Augmentation $\neq$ Synergy: The paper makes a crucial distinction. Augmentation is simply doing better with AI than without it. Synergy is doing better as a team than either the human or the AI could do alone.
  • The Synergy Gap: We are currently good at augmentation but bad at synergy. Human-AI teams frequently suffer from "performance losses," where the human actually drags down the performance of a superior AI, or the AI fails to elevate the human to its own level.
  • AI Strength as a Predictor: The biggest factor in whether AI helps a human is the AI's baseline ability. Augmentation is highest when the AI is significantly stronger than the human, particularly in decision-making tasks.
  • Process Matters: Synergy isn't just about having a better model; it's about how the work is organized. The analysis found that a clear division of labor (assigning different sub-tasks to humans vs. AI) and AI explanations were significant drivers of better collaborative outcomes.

Key Results

The quantitative results from the most recent run (synthesized in Figure 3) provide a sobering look at the state of collaboration:

  • Human Augmentation: The pooled effect size was $g = 0.53$ ($p < 0.001$). In 70% of the studied cases, the human-AI combination outperformed the human alone.
  • Human-AI Synergy: The pooled effect size was $g = -0.35$ ($p < 0.001$). In 64% of cases, the team performed worse than the best standalone performer.
  • Moderator Effects: Division of labor ($p = 0.031$) and providing AI explanations ($p = 0.002$) both significantly improved the chances of achieving synergy.
Figure 3: Forest plots of human augmentation and human–AI synergy.

Limitations and Future Work

The authors are transparent about the risks of an automated approach. LLM agents can misinterpret complex experimental designs or extract incorrect statistics. While their audit showed high agreement (83-90%), errors at the extraction stage can propagate through the entire analysis.

Furthermore, the "synergy" findings are based on a smaller subset of papers (171 effect sizes) that actually bothered to measure the AI's performance alone. Many researchers only compare "Human+AI" to "Human," which can mask the fact that the AI might have been better off working by itself. Future work will focus on scaling the audit process and encouraging researchers to adopt more rigorous three-way comparisons (Human vs. AI vs. Human+AI).

Summary updated

Abstract

As generative artificial intelligence (AI) becomes more capable and widely used, understanding when and how humans should work with these systems is becoming increasingly important. However, the empirical evidence in this space offers conflicting recommendations, which leaves individuals and organizations without clear guidance on when to combine human and AI capabilities. Existing meta-analyses can help reconcile these findings, but they require substantial manual effort and cannot readily incorporate new evidence as it emerges. To address this challenge, we introduce AutoAnalysis: an automated, self-updating meta-analysis of when humans and AI work better together, and when they work better alone. AutoAnalysis' most recent run synthesizes 1,955 effect sizes from 420 papers. Across this body of work, we found that, on average, human-AI combinations performed significantly better than humans alone, but they performed significantly worse than the better of humans or AI alone. These findings highlight the important distinction between improving human performance and achieving human-AI synergy and underscore the need to evaluate collaboration against both human and AI performance alone. As new studies emerge, AutoAnalysis will integrate their findings into the meta-analysis, providing cumulative evidence about when humans and AI perform better together and how human-AI performance varies across AI agents, tasks, and processes.
BibTeX
@misc{vaccaro2026autoanalysis,
  author = {Michelle Vaccaro and Abdullah Almaatouq and Thomas W. Malone},
  title = {{AutoAnalysis: An automated, self-updating meta-analysis of human-AI collaboration}},
  year = {2026},
  eprint = {2610.42001},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.42001},
}
Figure