TL;DR
InfoOrgBench is a new benchmark designed to test if AI agents can organize scattered records into accurate, multi-level hierarchies. The study reveals a significant "hierarchy gap": while modern LLMs are decent at clustering related items, they frequently fail to connect these groups correctly to higher-level parents, with model choice being far more critical than the agent framework used.
What Problem Does This Paper Address?
As AI agents take on more administrative roles—summarizing meetings, managing bug trackers, and organizing design documents—their ability to create useful folder structures becomes vital. However, organizing information isn't just about putting similar things together (clustering); it's about placing those groups into the correct broader context (hierarchy).
Existing benchmarks often focus on flat task completion or simple text clustering. They miss a subtle but critical failure mode: an agent might perfectly group all excerpts of a "Storage" proposal together but then mistakenly file that entire group under the "Networking" department. The authors argue that we need to evaluate parent-level relations separately from leaf-level grouping to truly understand an agent's organizational intelligence.
Methods
The authors introduce a rigorous evaluation framework centered on the Structure Score ($S$). This metric is designed to be "name-agnostic"—it doesn't care if you call a folder "Meeting 1" or "Project Alpha," only that the records inside are grouped correctly relative to each other.
Key components of the method include:
- Rooted Triples: A structural comparison method that checks if any three records ($a, b, c$) share the same relative hierarchy in the agent's output as they do in the ground truth.
- Level-Balanced F1: This ensures that errors at the top of the hierarchy (e.g., mixing up entire departments) aren't drowned out by the much more numerous relations at the bottom (e.g., individual files).
- Folder Economy: A penalty for "messy" agents that create unnecessary single-child folders.
- Agent Harnesses: The researchers tested 45 different systems by pairing nine state-of-the-art models (like GPT-6.1 Sol and Claude Sonnet 5.5) with five different agent "harnesses" (frameworks like ReAct-FC and Codex CLI) to see how much the environment affects performance.
Datasets
InfoOrgBench consists of 150 tasks across three distinct real-world domains:
- Recap (Meeting Summaries): Organizing sentences from the AMI meeting corpus by meeting and section.
- Thread (Issue Tracking): Grouping Jira subtasks and comments under their parent issues.
- Design (Technical Proposals): Organizing Kubernetes Enhancement Proposal (KEP) excerpts by their owning Special Interest Group (SIG) and specific proposal.
The benchmark is unique because its "ground truth" is derived deterministically from the original source structures, rather than subjective human labeling.
Key Insights and Ideas
- The "Hierarchy Gap": In every single domain, agents were significantly worse at recovering parent-level relations than leaf-level groups. For example, in the Design domain, agents recovered 99% of local proposal groups but failed on 13% of the parent SIG relations.
- Model > Harness: One of the most striking findings is that the specific LLM used accounts for nearly all the performance variance. Model means spanned 27 points, while the choice of agent harness (the "wrapper" around the model) changed the score by less than one point on average.
- Domain-Specific Failures:
- In Recap, agents often ignored the "source meeting" and grouped notes by topic instead.
- In Thread, agents frequently "flattened" the hierarchy, omitting the parent-issue layer entirely.
- Compute Isn't a Silver Bullet: Increasing "test-time compute" through higher reasoning effort or aggregating multiple runs did not significantly improve organizational accuracy. The problem seems to be one of inference (understanding how groups relate) rather than raw processing power.
Key Results
The quantitative results highlight that while we've made progress, autonomous organization is far from solved:
- Top Performer: GPT-6.1 Sol achieved the highest average Structure Score of 83.2/100.
- Difficulty Variance: No system exceeded a score of 78 on the Recap or Thread tasks, which require more nuanced inference of relationships. The Design tasks were "easier," with the best system reaching 95.8.
- Error Distribution: As shown in Figure 2, parent-level error rates (conflicting or unresolved triples) consistently dwarf leaf-level errors across all task families.
Limitations and Future Work
The authors acknowledge several limitations:
- Source Bias: The benchmark uses public English-language sources (Jira, Kubernetes, AMI), which might have been present in the models' training data.
- Subjectivity: The "correct" hierarchy is defined as the source's original structure. In reality, multiple ways of organizing the same data might be useful to a human user.
- Independence: Because tasks within a family reuse some source material, they aren't perfectly independent samples.
Future work could involve expanding to non-English datasets and conducting user studies to see if "incorrect" hierarchies (according to the source) might still be considered "useful" by humans.
Summary updated