InfoOrgBench: Can Agents Turn Scattered Information into Coherent Hierarchies?

Justice Ou, Lanming Ma, Zijian Zhang, Yilun ZhaoarXiv 2026 · arXiv

TL;DR

InfoOrgBench is a new benchmark designed to test if AI agents can organize scattered records into accurate, multi-level hierarchies. The study reveals a significant "hierarchy gap": while modern LLMs are decent at clustering related items, they frequently fail to connect these groups correctly to higher-level parents, with model choice being far more critical than the agent framework used.

What Problem Does This Paper Address?

As AI agents take on more administrative roles—summarizing meetings, managing bug trackers, and organizing design documents—their ability to create useful folder structures becomes vital. However, organizing information isn't just about putting similar things together (clustering); it's about placing those groups into the correct broader context (hierarchy).

Existing benchmarks often focus on flat task completion or simple text clustering. They miss a subtle but critical failure mode: an agent might perfectly group all excerpts of a "Storage" proposal together but then mistakenly file that entire group under the "Networking" department. The authors argue that we need to evaluate parent-level relations separately from leaf-level grouping to truly understand an agent's organizational intelligence.

Methods

The authors introduce a rigorous evaluation framework centered on the Structure Score ($S$). This metric is designed to be "name-agnostic"—it doesn't care if you call a folder "Meeting 1" or "Project Alpha," only that the records inside are grouped correctly relative to each other.

Key components of the method include:

  • Rooted Triples: A structural comparison method that checks if any three records ($a, b, c$) share the same relative hierarchy in the agent's output as they do in the ground truth.
  • Level-Balanced F1: This ensures that errors at the top of the hierarchy (e.g., mixing up entire departments) aren't drowned out by the much more numerous relations at the bottom (e.g., individual files).
  • Folder Economy: A penalty for "messy" agents that create unnecessary single-child folders.
  • Agent Harnesses: The researchers tested 45 different systems by pairing nine state-of-the-art models (like GPT-6.1 Sol and Claude Sonnet 5.5) with five different agent "harnesses" (frameworks like ReAct-FC and Codex CLI) to see how much the environment affects performance.
Figure 1: Overview of INFOORGBENCH. Agents see records with opaque IDs and an organizing criterion, and reorganize them through harness tools; the final files are captured when the agent stops. In this Design example, the agent keeps the immutable-secrets KEP intact but files it under the wrong SIG: the leaf-level triple agrees with the hidden reference, while the parent-level triple conflicts. Valid outputs are scored by the Structure Score S, which ignores folder names; E penalizes excess single-child folders. KEPs are Kubernetes Enhancement Proposals, grouped by their owning Special Interest Groups (SIGs).

Datasets

InfoOrgBench consists of 150 tasks across three distinct real-world domains:

  1. Recap (Meeting Summaries): Organizing sentences from the AMI meeting corpus by meeting and section.
  2. Thread (Issue Tracking): Grouping Jira subtasks and comments under their parent issues.
  3. Design (Technical Proposals): Organizing Kubernetes Enhancement Proposal (KEP) excerpts by their owning Special Interest Group (SIG) and specific proposal.

The benchmark is unique because its "ground truth" is derived deterministically from the original source structures, rather than subjective human labeling.

Comparison of InfoOrgBench with existing agent and hierarchy benchmarks.

Key Insights and Ideas

  • The "Hierarchy Gap": In every single domain, agents were significantly worse at recovering parent-level relations than leaf-level groups. For example, in the Design domain, agents recovered 99% of local proposal groups but failed on 13% of the parent SIG relations.
  • Model > Harness: One of the most striking findings is that the specific LLM used accounts for nearly all the performance variance. Model means spanned 27 points, while the choice of agent harness (the "wrapper" around the model) changed the score by less than one point on average.
  • Domain-Specific Failures:
    • In Recap, agents often ignored the "source meeting" and grouped notes by topic instead.
  • In Thread, agents frequently "flattened" the hierarchy, omitting the parent-issue layer entirely.
  • Compute Isn't a Silver Bullet: Increasing "test-time compute" through higher reasoning effort or aggregating multiple runs did not significantly improve organizational accuracy. The problem seems to be one of inference (understanding how groups relate) rather than raw processing power.

Key Results

The quantitative results highlight that while we've made progress, autonomous organization is far from solved:

  • Top Performer: GPT-6.1 Sol achieved the highest average Structure Score of 83.2/100.
  • Difficulty Variance: No system exceeded a score of 78 on the Recap or Thread tasks, which require more nuanced inference of relationships. The Design tasks were "easier," with the best system reaching 95.8.
  • Error Distribution: As shown in Figure 2, parent-level error rates (conflicting or unresolved triples) consistently dwarf leaf-level errors across all task families.
Figure 2: Error rates for parent-level and leaf-level re- lations. Parent-level error rates exceed leaf-level error rates in all three families. Recap errors are mainly con- flicting, Thread errors mainly unresolved, and Design errors occur almost entirely at the SIG level. Bars show the share of record triples distinguished by the reference that the output gets wrong, split into conflicting (the output groups a different pair together) and unresolved (the output does not separate the three records). Medi- ans over systems with per-level data (25 for Recap and Design, 21 for Thread). In parentheses: the share of outputs with at least one error at that level.
Detailed results of the 45 systems across the three task families.

Limitations and Future Work

The authors acknowledge several limitations:

  • Source Bias: The benchmark uses public English-language sources (Jira, Kubernetes, AMI), which might have been present in the models' training data.
  • Subjectivity: The "correct" hierarchy is defined as the source's original structure. In reality, multiple ways of organizing the same data might be useful to a human user.
  • Independence: Because tasks within a family reuse some source material, they aren't perfectly independent samples.

Future work could involve expanding to non-English datasets and conducting user studies to see if "incorrect" hierarchies (according to the source) might still be considered "useful" by humans.

Summary updated

Abstract

When agents organize records into folders, they may correctly group the excerpts of a design proposal yet place the proposal under the wrong owning group. Existing agent benchmarks do not directly measure such higher-level errors. We introduce InfoOrgBench, a benchmark of 150 tasks in which agents organize scattered records from meeting recaps, issue threads and design proposals into hierarchies. Integrity checks verify that every record is preserved, and a Structure Score measures agreement with the source hierarchy at every level, independent of folder names and order. We evaluate 45 systems that pair nine models with five agent harnesses. The best system scores 83.2 out of 100, but no system exceeds 78 on meeting recaps or issue threads. Model choice affects scores far more than harness choice: model means span about 27 points, whereas the average effect of each harness is below one point. More importantly, parent-level relations have higher error rates than leaf-level relations in all three domains. In design proposals, nearly half of the outputs analyzed by level make exactly this kind of error. These results show that evaluating information organization requires measuring higher-level relations separately from record preservation and local grouping.
BibTeX
@misc{ou2026infoorgbench,
  author = {Justice Ou and Lanming Ma and Zijian Zhang and Yilun Zhao},
  title = {{InfoOrgBench: Can Agents Turn Scattered Information into Coherent Hierarchies?}},
  year = {2026},
  eprint = {2611.10600},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10600},
}
Figure