Abstract
We introduce GAIA, a benchmark for General AI Assistants that, if solved, would represent a milestone in AI research. GAIA proposes real-world questions that require a set of fundamental abilities such as reasoning, multi-modality handling, web browsing, and generally tool-use proficiency. GAIA questions are conceptually simple for humans yet challenging for most advanced AIs: we show that human respondents obtain 92% vs. 15% for GPT-4 equipped with plugins. This notable performance disparity contrasts with the recent trend of LLMs outperforming humans on tasks requiring professional skills in e.g. law or chemistry. GAIA's philosophy departs from the current trend in AI benchmarks suggesting to target tasks that are ever more difficult for humans. We posit that the advent of Artificial General Intelligence (AGI) hinges on a system's capability to exhibit similar robustness as the average human does on such questions. Using GAIA's methodology, we devise 466 questions and their answer. We release our questions while retaining answers to 300 of them to power a leader-board available at https://huggingface.co/gaia-benchmark.
Introduction: The "Benchmark Saturation" Crisis
Large Language Models (LLMs) are currently breaking records on academic benchmarks like MMLU or GSM8k at an unprecedented rate. However, these benchmarks often focus on tasks that are difficult for humans—such as complex legal reasoning or advanced STEM questions—which AI can often solve through memorization or data contamination. We are missing a metric for Artificial General Intelligence (AGI) that reflects the robustness of an average human in everyday scenarios. GAIA (General AI Assistants) addresses this by shifting the focus to "conceptually simple" tasks that require multi-step reasoning and tool use.
The GAIA Benchmark Design
GAIA consists of 466 questions carefully crafted by humans to be "non-gameable." The answers are factoid (numbers or short strings) and cannot be found in plain text on the internet; solving them requires active planning and execution. The benchmark is structured into three levels of increasing complexity:
- Level 1: Generally requires no tools or just one tool, involving no more than 5 steps.
- Level 2: Involves 5 to 10 steps and requires combining different tools.
- Level 3: Designed for a "near-perfect" assistant, requiring arbitrarily long sequences of actions and multiple tools.
Methodology and Required Capabilities
To solve GAIA questions, an AI must be able to orchestrate various tools and handle diverse data formats. The tasks often involve multi-modal attachments like .xlsx, .pdf, .png, or .mp3 files. The primary hurdle for current models is not raw intelligence, but the ability to plan and use tools like web browsers and code interpreters over long sequences without losing track of the goal.
Key Results: The Performance Inversion
The disparity between human and AI performance is stark. While human respondents achieved a 92% success rate across all levels, GPT-4 (equipped with plugins) only achieved about 15% overall. Even on the easiest Level 1 tasks, the top-performing model peaked at 30%.
Other autonomous systems like AutoGPT performed poorly, reaching around 14% on Level 1 and nearly 0% on Level 2, often proving slower and less reliable than manual tool selection. Notably, Level 3 remains a "zero zone," with no current AI system able to solve any questions in this category. This "performance inversion" highlights the long road ahead for true general-purpose assistants.
Conclusion
GAIA serves as a reality check for the AI industry. It suggests that while LLMs are brilliant scholars, they remain clumsy assistants. The path to AGI requires systems that can exhibit the same robustness and tool-use proficiency as an average human when faced with real-world, multi-step problems.
Summary updated