TL;DR
AutoAnalysis is an LLM-powered pipeline that automates the systematic review process to keep pace with the exponential growth of human-AI collaboration research. Its most striking finding is a persistent "synergy gap": while AI consistently helps humans perform better than they would alone, these teams usually underperform compared to the best standalone agent—whether that’s the human or the AI.
What Problem Does This Paper Address?
As generative AI becomes a staple in the workplace, researchers are flooding the literature with studies on how humans and AI should work together. However, the results are often contradictory—some studies show massive productivity gains, while others highlight significant performance dips.
Traditionally, we rely on meta-analyses to reconcile these conflicting findings. The problem? Manual meta-analyses are incredibly labor-intensive and are often outdated by the time they are published. Given the current explosion of research in this field, a static review is no longer sufficient.
Methods
The authors introduce AutoAnalysis, a "living" meta-analysis pipeline that uses frontier AI agents to automate every stage of a systematic review.
The pipeline follows a pre-registered protocol:
- Discovery: It starts with a "seed" of papers and uses a "snowballing" technique—automatically searching for forward and backward citations via the OpenAlex API.
- Screening & Extraction: Specialized LLM agents (specifically GPT-6.1-sol and GPT-6-astra) screen abstracts, read full-text PDFs, and extract quantitative data (means, SDs, sample sizes) and moderator variables (task type, AI explanations, etc.).
- Synthesis: The system employs a three-level random-effects model to calculate pooled effect sizes while accounting for the fact that multiple results often come from the same paper.
- Validation: To ensure accuracy, the authors implemented "agent audits" where a different model family (Claude Fable 5.1) re-evaluates a sample of the primary agents' decisions, alongside manual human sanity checks.
Datasets
AutoAnalysis doesn't use a fixed training dataset; instead, it operates on a dynamic corpus of experimental literature. As of October 2026, the system had processed:
- 15,495 unique records identified through snowballing.
- 420 papers meeting strict inclusion criteria (original experiments with human participants and generative AI).
- 1,955 unique effect sizes synthesized in the final model.
Key Insights and Ideas
- Augmentation $\neq$ Synergy: The paper makes a crucial distinction. Augmentation is simply doing better with AI than without it. Synergy is doing better as a team than either the human or the AI could do alone.
- The Synergy Gap: We are currently good at augmentation but bad at synergy. Human-AI teams frequently suffer from "performance losses," where the human actually drags down the performance of a superior AI, or the AI fails to elevate the human to its own level.
- AI Strength as a Predictor: The biggest factor in whether AI helps a human is the AI's baseline ability. Augmentation is highest when the AI is significantly stronger than the human, particularly in decision-making tasks.
- Process Matters: Synergy isn't just about having a better model; it's about how the work is organized. The analysis found that a clear division of labor (assigning different sub-tasks to humans vs. AI) and AI explanations were significant drivers of better collaborative outcomes.
Key Results
The quantitative results from the most recent run (synthesized in Figure 3) provide a sobering look at the state of collaboration:
- Human Augmentation: The pooled effect size was $g = 0.53$ ($p < 0.001$). In 70% of the studied cases, the human-AI combination outperformed the human alone.
- Human-AI Synergy: The pooled effect size was $g = -0.35$ ($p < 0.001$). In 64% of cases, the team performed worse than the best standalone performer.
- Moderator Effects: Division of labor ($p = 0.031$) and providing AI explanations ($p = 0.002$) both significantly improved the chances of achieving synergy.
Limitations and Future Work
The authors are transparent about the risks of an automated approach. LLM agents can misinterpret complex experimental designs or extract incorrect statistics. While their audit showed high agreement (83-90%), errors at the extraction stage can propagate through the entire analysis.
Furthermore, the "synergy" findings are based on a smaller subset of papers (171 effect sizes) that actually bothered to measure the AI's performance alone. Many researchers only compare "Human+AI" to "Human," which can mask the fact that the AI might have been better off working by itself. Future work will focus on scaling the audit process and encouraging researchers to adopt more rigorous three-way comparisons (Human vs. AI vs. Human+AI).
Summary updated