Annotrace: A Workbench for Step-Level Annotation of Agent Traces

Arun Josephraj Arokiaraj, Chenyi Huang, Sahan Bulathwela, Zekun Wu, Bin Wu

TL;DR

Annotrace is a specialized workbench designed for the fine-grained, step-level annotation of LLM agent traces. By bridging the gap between raw observability data and structured human labeling, it allows researchers to pinpoint exactly where a multi-agent workflow went wrong, even in runs that appear successful at the surface level.

What Problem Does This Paper Address?

When an LLM agent fails, researchers usually resort to manual, "throwaway" labeling tools to figure out why. While observability platforms (like LangSmith or Langfuse) are excellent at recording traces, they lack structured labeling features. Conversely, traditional annotation tools (like Label Studio) are designed for text spans or documents, not the nested, hierarchical nature of agentic steps—such as a coordinator delegating a task to a worker who then initiates a tool-call loop.

The authors argue that trace-level labeling (simply marking a run as "Success" or "Failure") is insufficient. A run might reach the correct final answer despite multiple internal errors that were luckily recovered. Without step-level annotation, these "silent" failures remain invisible, hindering the development of robust agent taxonomies.

Methods

Annotrace introduces a structured workflow for "joint human–AI" annotation:

  1. Adapters: The tool includes adapters for six major trace formats, including OpenTelemetry GenAI, LangSmith, and Langfuse. These adapters convert diverse logs into a unified, nested list of "Step" objects (decisions, tool calls, replies, etc.), preserving the original delegation depth.
  2. Schema as Questions: Instead of a rigid taxonomy, users define a schema as a list of questions (e.g., "Was this step correct?"). These can include conditional logic (e.g., only ask "What went wrong?" if the previous answer was "No").
  3. Model-in-the-Loop: A frontier LLM acts as a "proposer." It reads a sample of traces to suggest a starting schema and then pre-labels the steps. The human annotator then reviews, agrees, or disagrees with the model's suggestions.
  4. Agreement Metrics: The workbench built-in support for measuring inter-annotator agreement (Cohen’s $\kappa$), allowing researchers to compare human-human, human-AI, or AI-AI consistency.
Figure 2: Top: six trace formats, one adapter each, one nested step model. A step may contain finer steps, so nesting sur- vives the conversion. Bottom: the annotator writes the ques- tions; the model proposes a first set from a sample of runs, and later answers a run before the annotator does. Grey marks the model. Both the questions a person sees and the instruc- tions the model receives are generated from the same object, so the two cannot diverge.

Datasets

The authors evaluated Annotrace using a 40-run multi-agent corpus based on HotpotQA, a dataset for diverse, explainable multi-hop question answering. They used this corpus to demonstrate how the tool can reproduce manual labeling efforts and measure model agreement.

Key Insights and Ideas

  • Nesting is Critical: Flattening a trace into a simple transcript destroys the structural context a labeller needs. Annotrace preserves the hierarchy, making it clear which agent at which depth made a specific decision.
  • Recovering Buried Cycles: Some trace formats bury the agent's internal "think-search-respond" loop in a single text field. Annotrace's adapters are designed to recover these cycles so they can be labeled individually.

    Figure 3: Where a format buries the agent loop in one text field, the adapter recovers Thinking-Action-Response cycles, each labelled separately. This reproduces 235 units of a pre- vious manual effort exactly.
  • The "Correct" Run Fallacy: As shown in Figure 1 of the paper, a run can be marked "correct" at the trace level while containing eight wasted search cycles and internal reasoning contradictions. Step-level annotation is the only way to surface these inefficiencies.

  • Dynamic Schema Induction: By letting the model propose labels based on a sample of the actual data, researchers can build taxonomies that are grounded in the specific failures of their system rather than trying to fit their data into a generic, pre-existing framework.

Key Results

  • Precision: On the HotpotQA corpus, the coordinator adapter reproduced all 235 Thinking-Action-Response unit keys from a previous manual effort with 100% accuracy.
  • Consistency: Two different models labeling the same traces agreed 91% of the time ($\kappa = 0.82$), suggesting that AI-assisted labeling is highly reliable for initial passes.
  • Efficiency: Generating a schema proposal takes roughly 4 seconds, and annotating a single run takes between 3 to 5 seconds at a negligible cost (fractions of a cent per call).
  • User Experience: The interface (built with Streamlit) keeps the global context (question/final answer) in view while the user scrolls through individual steps, significantly reducing cognitive load.
Figure 4: The annotation screen. The sidebar holds the run’s question and the agent’s answer, so both stay in view while the steps scroll. Colour marks the kind of step until a label is chosen, then marks the label. Steps still wanting an answer say so.

Limitations and Future Work

The authors note that the granularity of the annotation is ultimately limited by the input format; if a trace format doesn't record a specific internal state, Annotrace cannot invent it. While the tool currently supports six formats, the ecosystem is evolving rapidly, requiring constant adapter updates. Future work may involve deeper integration with automated failure detection systems like AgentGraph to further streamline the discovery of problematic steps.

Summary updated

Abstract

Every study of why LLM agents fail begins by labelling traces by hand, and every such study builds its own throwaway tool. Observability platforms record agent traces but offer no structured labelling; annotation tools label text but have no notion of a step, a delegation or a tool call, leaving it unclear what the smallest unit of execution to label should be. Annotrace reads six trace formats, including OpenTelemetry GenAI, LangSmith and Langfuse, and turns each run into an ordered, nested list of steps. The annotator writes their own labelling codebook; a frontier model proposes a starting set from a sample of their own traces, answers first, and is asked exactly the questions the human is asked, making annotation a joint human-AI process. Annotrace measures agreement between any two annotators, human or AI. On a multi-agent question-answering corpus it reproduces a previous manual effort's 235 annotation units exactly, and two models labelling the same traces agree 91% of the time at $κ= 0.82$.
BibTeX
@misc{arokiaraj2026annotrace,
  author = {Arun Josephraj Arokiaraj and Chenyi Huang and Sahan Bulathwela and Zekun Wu and Bin Wu},
  title = {{Annotrace: A Workbench for Step-Level Annotation of Agent Traces}},
  year = {2026},
  eprint = {2610.23284},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.23284},
}
Figure