TL;DR
iAgentBench is a dynamic open-domain question answering (ODQA) benchmark designed to evaluate how well AI agents can synthesize and make sense of information spread across multiple web sources. By generating questions from real-world, high-traffic topics that strictly require connecting different thematic "stories," it reveals that while search tools give large language models (LLMs) better access to facts, integrating those facts to answer complex queries remains a major bottleneck.
What Problem Does This Paper Address?
As search-enabled LLMs and information-seeking agents (ISAs) become ubiquitous, we rely on them to browse, aggregate, and reconcile information on our behalf. However, existing evaluation benchmarks fail to test this properly. Many popular datasets are "passage-centric," meaning a question can be answered by extracting a single snippet from a single retrieved document.
This creates a massive blind spot: it doesn't measure sensemaking—the ability to connect distributed evidence, track causal links, and resolve dependencies across different facets of a topic. Furthermore, static benchmarks quickly become obsolete or suffer from data contamination as newer models memorize them during pre-training.
Methods
The authors introduce a pipeline to dynamically construct iAgentBench using a four-step process:
- Interest-Driven Seeds: Instead of using static trivia lists, the pipeline pulls real-time, high-traffic topics from public attention signals via the GDELT Global Knowledge Graph.
- Graph Construction: The system runs a web search on a topic, collects the top results, and uses an LLM to extract entities and natural-language claims. These claims form a "story graph," which is then clustered into distinct thematic communities using the Leiden algorithm.
- Role Assignment: Communities are evaluated using network centrality metrics (like PageRank and betweenness) to assign them roles: Core (dominant sub-stories), Bridge (themes connecting separate stories), or Satellite (supporting context).
- QA Generation & Verification: The pipeline packages connected themes into compact "packets" linked by explicit cross-community connector edges. An LLM generates natural, user-like questions based on specific intent patterns (e.g., triggers, consequences, connections). Finally, a panel of three LLM judges filters out low-quality or single-passage questions to ensure cross-theme synthesis is strictly necessary.
Datasets
The paper introduces iAgentBench, a refreshable, auditable benchmark for open-domain QA. For evaluation, the authors compare LLM performance on iAgentBench against two prominent benchmarks:
- SimpleQA: A benchmark testing short-form factuality and precise extraction.
- HotpotQA: A classic multi-hop question-answering dataset.
Key Insights and Ideas
- Access $\neq$ Integration: Providing models with search results (Retrieval-Augmented Generation, or RAG) yields massive accuracy spikes on traditional fact-lookup datasets like SimpleQA. However, on iAgentBench, a substantial performance gap remains, proving that retrieving evidence is only half the battle; synthesizing it is the real challenge.
- Agentic Iteration Can Backfire: Multi-step agentic workflows (like self-reflection via Reflexion) are not a silver bullet. While extra reasoning steps help some models connect dispersed evidence, they cause others to suffer from "reasoning drift" or over-correction, leading to performance regressions.
- The Value of Auditability: Because iAgentBench saves intermediate artifacts (retrieved documents, story-graph links, and community cards), researchers can precisely diagnose whether an agent failed because it couldn't find the right documents or because it couldn't connect the dots during synthesis.
Key Results
The authors evaluated four prominent LLMs (Claude Sonnet 4.5, LLaMA 4 Maverick 17B, Mistral Large 3, and Gemma 3 27B) across three settings: Base (no tools), RAG, and Reflexion.
- RAG Performance: While RAG closed the difficulty gap for SimpleQA (making it the easiest benchmark for most models once text was retrieved), iAgentBench remained consistently difficult. For example, Claude Sonnet 4.5 achieved a 0.740 accuracy on SimpleQA with RAG, but only 0.648 on iAgentBench.
- Agentic Gains vs. Losses: As shown in the decomposition of improvements, LLaMA benefited significantly from agentic self-reflection (+0.096), whereas Mistral Large 3 (-0.074) and Gemma 3 (-0.022) saw performance drops when forced to iterate on iAgentBench queries.
Limitations and Future Work
- High Construction Cost: Because the benchmark is dynamic and refreshable, it incurs notable compute and API overhead from web searching, graph extraction, and running multi-LLM judge panels for every new evaluation window.
- Dependence on Generative Pipelines: The pipeline relies on LLMs to build the underlying story graphs and verify the questions, introducing risks of algorithmic bias or omissions.
- Future Work: The authors suggest expanding the benchmark to evaluate alternative retrieval mechanisms, longer-horizon agent planning strategies, and broader hyperparameter configurations.
Summary updated