How to Interpret Agent Behavior

Jie Gao, Kaiser Sun, Jen-tse Huang, Katherine Van Koevering, Sijie Ji, Heyuan Huang, Weiyan Shi, Zhuoran Lu, Ziang Xiao, Daniel Khashabi, Mark DredzearXiv 2026 · arXiv

TL;DR

As LLM-based agents tackle increasingly complex, multi-hour tasks, understanding their messy, natural-language execution traces has become a major bottleneck. This paper introduces Act·ONOMY, a standardized, three-level hierarchical taxonomy of agent behaviors, paired with an automated tool to parse raw trajectories into structured, human-readable behavioral profiles. This framework enables researchers to systematically diagnose agent failures, optimize workflows, and maintain meaningful runtime oversight.

What Problem Does This Paper Address?

Modern AI agents (like Claude Code or SWE-agent) no longer just answer single prompts; they operate autonomously for hours or days, navigating codebases, web interfaces, and scientific tools. When these agents fail, they rarely do so cleanly—they loop, recover, hallucinate, or hit dead ends before reaching a final outcome.

Currently, the primary way to understand this "runtime behavior" is by reading raw reasoning trajectories (the logs of thoughts and actions). However, these logs are unstructured, verbose, and incredibly tedious for humans to analyze at scale. Furthermore, the AI research community lacks a shared, standardized vocabulary to describe what agents are actually doing. Without a common language, it is impossible to compare agent architectures systematically, perform behavioral regression testing, or analyze failure patterns across different systems.

Methods

The authors developed Act·ONOMY using a Grounded Theory approach, anchoring it in the established Cognitive Architectures for Language Agents (CoALA) framework.

  1. Taxonomy Construction: They extracted behavior descriptions from recent AI literature, normalized them into a uniform "verb + noun" format, and iteratively scaled the codebook using an LLM-assisted qualitative analysis pipeline reviewed by human experts.
  2. Hierarchical Structure: The final taxonomy organizes agent behaviors into three high-level cognitive clusters: External Interaction, Cognition & Execution, and Learning & Adaptation. It spans 10 main actions, 46 subactions, and 120 leaf categories.
Figure 3: An at-a-glance map of Act·ONOMY.
  1. Automated-Trace-Analysis-Tool: To apply this taxonomy at scale, the authors built an automated pipeline that takes raw agent trajectories from any framework, parses them into per-turn triples (Observation $\rightarrow$ Thought $\rightarrow$ Action), extracts behavioral indicators, and maps them to Act·ONOMY tags to generate structured behavioral profiles.

Datasets

The paper utilizes three distinct data sources for construction, validation, and case studies:

  • Construction Corpus: 565 behavior-description sentences manually verified from 35 peer-reviewed agent papers (NeurIPS, ICML, ICLR, and ACL 2024–2026).
  • Generalization Dataset: 3,455 behavioral descriptions automatically extracted from 211 agent-related papers curated from the awesome-language-agents GitHub repository to test the taxonomy's coverage.
  • Trajectory Evaluation Traces: 300 public execution traces from three distinct agent frameworks (AG2, HyperAgent, and SWE-Agent), alongside two specific deep-dive trajectories from the SWE-bench benchmark.

Key Insights and Ideas

  • Thinking vs. Acting Bias: Large-scale analysis of the literature revealed that current research heavily emphasizes agent cognition over physical execution. Reasoning and Grounding make up over 40% of all behavioral descriptions, while Executing accounts for less than 2%, suggesting that papers treat the actual act of execution as incidental.
  • Top-Heavy but Long-Tailed: Agent behaviors are highly concentrated at the top action level but incredibly diverse at the leaf level. The top 10 leaf codes each account for less than 6% of the total dataset, showing that fine-grained agent actions are highly varied.
Action co-occurrence matrix
  • Surfacing Invisible Failure Modes: Standard outcome metrics (like success rate) only tell you if an agent failed, not why. By using fine-grained leaf categories, Act·ONOMY successfully exposed subtle, recurring failure patterns—such as a "submit anyway, without verifying" loop where an agent explicitly notes it cannot verify a fix due to environment constraints but proceeds to submit it regardless.

Key Results

  • High Reliability: The taxonomy achieved strong inter-rater reliability, with a human-to-human Cohen’s $\kappa$ of 0.87 at the action level and 0.72 at the subaction level. An LLM deductive analyst closely mirrored human coding ($\kappa = 0.74$ at the action level).
  • Distinct Agent Fingerprints: The automated tool successfully mapped distinct behavioral profiles that reflect different agent architectures. For instance, AG2 (focused on math) scored significantly higher on Evaluating and Deciding, while HyperAgent (a multi-agent software engineering system) leaned heavily into Reflecting and Memory.
Behavioral profiles of AG2, HyperAgent, and SWE-Agent

Limitations and Future Work

  • Literature Bias: Because the taxonomy was built from what researchers write about their agents, it captures high-level, aspirational architectural moves and might under-represent silent, low-level execution quirks that only appear in raw logs.
  • Annotation Dependency: The current Automated-Trace-Analysis-Tool relies heavily on commercial LLMs (specifically Claude Opus), introducing potential API costs and reproducibility challenges as models are retired or updated.
  • Demographic Skew: The source papers are drawn from top-tier Western AI venues, meaning behaviors documented in regional venues, industry reports, or non-English literature are currently under-represented. The authors plan to treat Act·ONOMY as a "living taxonomy" hosted on GitHub to allow community-driven extensions.

Summary updated

Abstract

Autonomous agents such as Claude Code and Codex now operate for hours or even days. Understanding their runtime behavior has become critical for downstream tasks such as diagnosing inefficiencies, fixing bugs, and ensuring better oversight. A primary way to gain this understanding is analyzing the reasoning trajectories and execution traces these agents generate. Yet such data remains in unstructured natural-language form, making it difficult for humans to interpret at scale. We introduce ACT*ONOMY (a combination of Action and Taxonomy), a taxonomy for describing and analyzing agent behavior at runtime. ACT*ONOMY has two components: (1) the taxonomy itself, developed through Grounded Theory and structured as a three-level hierarchy of 10 actions, 46 subactions, and 120 leaf categories; and (2) an open repository that hosts the living taxonomy, provides an automated analysis pipeline that applies it to agent trajectories analysis, and defines an extension protocol for customization and growth. Our experiments show that ACTONOMY can compare behavioral profiles across agents and characterize a single agent's behavior across diverse trajectories, surfacing patterns indicative of failure modes. By providing a shared vocabulary, ACT*ONOMY helps researchers, agent designers, and end users interpret agent behavior more consistently, enabling better oversight and control.
BibTeX
@misc{gao2026how,
  author = {Jie Gao and Kaiser Sun and Jen-tse Huang and Katherine Van Koevering and Sijie Ji and Heyuan Huang and Weiyan Shi and Zhuoran Lu and Ziang Xiao and Daniel Khashabi and Mark Dredze},
  title = {{How to Interpret Agent Behavior}},
  year = {2026},
  eprint = {2605.13625},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.13625},
}

Loading related content...

Figure