TL;DR
As LLM-based agents tackle increasingly complex, multi-hour tasks, understanding their messy, natural-language execution traces has become a major bottleneck. This paper introduces Act·ONOMY, a standardized, three-level hierarchical taxonomy of agent behaviors, paired with an automated tool to parse raw trajectories into structured, human-readable behavioral profiles. This framework enables researchers to systematically diagnose agent failures, optimize workflows, and maintain meaningful runtime oversight.
What Problem Does This Paper Address?
Modern AI agents (like Claude Code or SWE-agent) no longer just answer single prompts; they operate autonomously for hours or days, navigating codebases, web interfaces, and scientific tools. When these agents fail, they rarely do so cleanly—they loop, recover, hallucinate, or hit dead ends before reaching a final outcome.
Currently, the primary way to understand this "runtime behavior" is by reading raw reasoning trajectories (the logs of thoughts and actions). However, these logs are unstructured, verbose, and incredibly tedious for humans to analyze at scale. Furthermore, the AI research community lacks a shared, standardized vocabulary to describe what agents are actually doing. Without a common language, it is impossible to compare agent architectures systematically, perform behavioral regression testing, or analyze failure patterns across different systems.
Methods
The authors developed Act·ONOMY using a Grounded Theory approach, anchoring it in the established Cognitive Architectures for Language Agents (CoALA) framework.
- Taxonomy Construction: They extracted behavior descriptions from recent AI literature, normalized them into a uniform "verb + noun" format, and iteratively scaled the codebook using an LLM-assisted qualitative analysis pipeline reviewed by human experts.
- Hierarchical Structure: The final taxonomy organizes agent behaviors into three high-level cognitive clusters: External Interaction, Cognition & Execution, and Learning & Adaptation. It spans 10 main actions, 46 subactions, and 120 leaf categories.
- Automated-Trace-Analysis-Tool: To apply this taxonomy at scale, the authors built an automated pipeline that takes raw agent trajectories from any framework, parses them into per-turn triples (Observation $\rightarrow$ Thought $\rightarrow$ Action), extracts behavioral indicators, and maps them to Act·ONOMY tags to generate structured behavioral profiles.
Datasets
The paper utilizes three distinct data sources for construction, validation, and case studies:
- Construction Corpus: 565 behavior-description sentences manually verified from 35 peer-reviewed agent papers (NeurIPS, ICML, ICLR, and ACL 2024–2026).
- Generalization Dataset: 3,455 behavioral descriptions automatically extracted from 211 agent-related papers curated from the awesome-language-agents GitHub repository to test the taxonomy's coverage.
- Trajectory Evaluation Traces: 300 public execution traces from three distinct agent frameworks (AG2, HyperAgent, and SWE-Agent), alongside two specific deep-dive trajectories from the SWE-bench benchmark.
Key Insights and Ideas
- Thinking vs. Acting Bias: Large-scale analysis of the literature revealed that current research heavily emphasizes agent cognition over physical execution. Reasoning and Grounding make up over 40% of all behavioral descriptions, while Executing accounts for less than 2%, suggesting that papers treat the actual act of execution as incidental.
- Top-Heavy but Long-Tailed: Agent behaviors are highly concentrated at the top action level but incredibly diverse at the leaf level. The top 10 leaf codes each account for less than 6% of the total dataset, showing that fine-grained agent actions are highly varied.
- Surfacing Invisible Failure Modes: Standard outcome metrics (like success rate) only tell you if an agent failed, not why. By using fine-grained leaf categories, Act·ONOMY successfully exposed subtle, recurring failure patterns—such as a "submit anyway, without verifying" loop where an agent explicitly notes it cannot verify a fix due to environment constraints but proceeds to submit it regardless.
Key Results
- High Reliability: The taxonomy achieved strong inter-rater reliability, with a human-to-human Cohen’s $\kappa$ of 0.87 at the action level and 0.72 at the subaction level. An LLM deductive analyst closely mirrored human coding ($\kappa = 0.74$ at the action level).
- Distinct Agent Fingerprints: The automated tool successfully mapped distinct behavioral profiles that reflect different agent architectures. For instance, AG2 (focused on math) scored significantly higher on Evaluating and Deciding, while HyperAgent (a multi-agent software engineering system) leaned heavily into Reflecting and Memory.
Limitations and Future Work
- Literature Bias: Because the taxonomy was built from what researchers write about their agents, it captures high-level, aspirational architectural moves and might under-represent silent, low-level execution quirks that only appear in raw logs.
- Annotation Dependency: The current Automated-Trace-Analysis-Tool relies heavily on commercial LLMs (specifically Claude Opus), introducing potential API costs and reproducibility challenges as models are retired or updated.
- Demographic Skew: The source papers are drawn from top-tier Western AI venues, meaning behaviors documented in regional venues, industry reports, or non-English literature are currently under-represented. The authors plan to treat Act·ONOMY as a "living taxonomy" hosted on GitHub to allow community-driven extensions.
Summary updated