iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics

Preetam Prabhu Srikar Dammu, Arnav Palkhiwala, Tanya G. Roosta, Chirag Shah

TL;DR

iAgentBench is a dynamic open-domain question answering (ODQA) benchmark designed to evaluate how well AI agents can synthesize and make sense of information spread across multiple web sources. By generating questions from real-world, high-traffic topics that strictly require connecting different thematic "stories," it reveals that while search tools give large language models (LLMs) better access to facts, integrating those facts to answer complex queries remains a major bottleneck.

What Problem Does This Paper Address?

As search-enabled LLMs and information-seeking agents (ISAs) become ubiquitous, we rely on them to browse, aggregate, and reconcile information on our behalf. However, existing evaluation benchmarks fail to test this properly. Many popular datasets are "passage-centric," meaning a question can be answered by extracting a single snippet from a single retrieved document.

This creates a massive blind spot: it doesn't measure sensemaking—the ability to connect distributed evidence, track causal links, and resolve dependencies across different facets of a topic. Furthermore, static benchmarks quickly become obsolete or suffer from data contamination as newer models memorize them during pre-training.

Methods

The authors introduce a pipeline to dynamically construct iAgentBench using a four-step process:

  1. Interest-Driven Seeds: Instead of using static trivia lists, the pipeline pulls real-time, high-traffic topics from public attention signals via the GDELT Global Knowledge Graph.
  2. Graph Construction: The system runs a web search on a topic, collects the top results, and uses an LLM to extract entities and natural-language claims. These claims form a "story graph," which is then clustered into distinct thematic communities using the Leiden algorithm.
  3. Role Assignment: Communities are evaluated using network centrality metrics (like PageRank and betweenness) to assign them roles: Core (dominant sub-stories), Bridge (themes connecting separate stories), or Satellite (supporting context).
  4. QA Generation & Verification: The pipeline packages connected themes into compact "packets" linked by explicit cross-community connector edges. An LLM generates natural, user-like questions based on specific intent patterns (e.g., triggers, consequences, connections). Finally, a panel of three LLM judges filters out low-quality or single-passage questions to ensure cross-theme synthesis is strictly necessary.
Overview of the iAgentBench pipeline

Datasets

The paper introduces iAgentBench, a refreshable, auditable benchmark for open-domain QA. For evaluation, the authors compare LLM performance on iAgentBench against two prominent benchmarks:

  • SimpleQA: A benchmark testing short-form factuality and precise extraction.
  • HotpotQA: A classic multi-hop question-answering dataset.

Key Insights and Ideas

  • Access $\neq$ Integration: Providing models with search results (Retrieval-Augmented Generation, or RAG) yields massive accuracy spikes on traditional fact-lookup datasets like SimpleQA. However, on iAgentBench, a substantial performance gap remains, proving that retrieving evidence is only half the battle; synthesizing it is the real challenge.
  • Agentic Iteration Can Backfire: Multi-step agentic workflows (like self-reflection via Reflexion) are not a silver bullet. While extra reasoning steps help some models connect dispersed evidence, they cause others to suffer from "reasoning drift" or over-correction, leading to performance regressions.
  • The Value of Auditability: Because iAgentBench saves intermediate artifacts (retrieved documents, story-graph links, and community cards), researchers can precisely diagnose whether an agent failed because it couldn't find the right documents or because it couldn't connect the dots during synthesis.

Key Results

The authors evaluated four prominent LLMs (Claude Sonnet 4.5, LLaMA 4 Maverick 17B, Mistral Large 3, and Gemma 3 27B) across three settings: Base (no tools), RAG, and Reflexion.

Base vs. RAG accuracy across datasets
  • RAG Performance: While RAG closed the difficulty gap for SimpleQA (making it the easiest benchmark for most models once text was retrieved), iAgentBench remained consistently difficult. For example, Claude Sonnet 4.5 achieved a 0.740 accuracy on SimpleQA with RAG, but only 0.648 on iAgentBench.
  • Agentic Gains vs. Losses: As shown in the decomposition of improvements, LLaMA benefited significantly from agentic self-reflection (+0.096), whereas Mistral Large 3 (-0.074) and Gemma 3 (-0.022) saw performance drops when forced to iterate on iAgentBench queries.
Retrieval gains vs. agentic gains

Limitations and Future Work

  • High Construction Cost: Because the benchmark is dynamic and refreshable, it incurs notable compute and API overhead from web searching, graph extraction, and running multi-LLM judge panels for every new evaluation window.
  • Dependence on Generative Pipelines: The pipeline relies on LLMs to build the underlying story graphs and verify the questions, introducing risks of algorithmic bias or omissions.
  • Future Work: The authors suggest expanding the benchmark to evaluate alternative retrieval mechanisms, longer-horizon agent planning strategies, and broader hyperparameter configurations.

Summary updated

Abstract

With the emergence of search-enabled generative QA systems, users are increasingly turning to tools that browse, aggregate, and reconcile evidence across multiple sources on their behalf. Yet many widely used QA benchmarks remain answerable by retrieving a single relevant passage, making them poorly suited for measuring cross-source sensemaking, such as integrating evidence, tracking causal links, and resolving dependencies across facets of a topic. We present iAgentBench, a dynamic ODQA benchmark that targets these higher-level information needs while keeping questions natural and grounded in realistic information-seeking behavior. iAgentBench draws seed topics from real-world attention signals and uses common user intent patterns to construct user-like questions whose answers require combining evidence from multiple sources, not just extracting a single snippet. Each instance is released with traceable evidence and auditable intermediate artifacts that support contamination checks and enable fine-grained diagnosis of failures in retrieval versus synthesis. Experiments across multiple LLMs show that retrieval improves accuracy, but retrieval alone does not reliably resolve these questions, underscoring the need to evaluate evidence use, not just evidence access.
BibTeX
@inproceedings{dammu2026iagentbench,
  author = {Preetam Prabhu Srikar Dammu and Arnav Palkhiwala and Tanya G. Roosta and Chirag Shah},
  title = {{iAgentBench: Benchmarking Sensemaking Capabilities of Information-Seeking Agents on High-Traffic Topics}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2026},
  doi = {10.1145/3805712.3808627},
}
Figure