Measuring and Reducing Unusable Replies of LLM Agents

Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu, Yutao YuearXiv 2026 · arXiv

TL;DR

Even when LLM agents successfully complete a task, their final reports are often unusable because they reference internal notes, tool outputs, or "coined" jargon that the user never saw. This paper introduces reply usability as a new evaluation metric, finding that 54% of agent replies fail this test, and demonstrates that simple prompting requirements or a secondary "rewriter" model can significantly improve clarity.

What Problem Does This Paper Address?

We are increasingly delegating complex, multi-step tasks to LLM agents. These agents act as "black boxes": they might call dozens of tools, read hundreds of files, and maintain internal scratchpads to track their progress. However, the human requester only sees the final reply.

The authors identify a major gap in current AI evaluation: benchmarks usually measure task success (did the agent find the answer?) or faithfulness (did the agent lie?). They ignore usability—whether the user can actually understand the report without having to dig through the agent's private logs. For example, if an agent tells you "Result R07 is better than R09," but you don't know what R07 and R09 refer to because they were temporary IDs generated during a tool call, the reply is technically accurate but practically useless.

An example of a failed reply where the agent references internal IDs (R07, R09) and configuration keys (reader_size: 70B) that the user has never seen.

Methods

The researchers defined a reply as usable if it meets three criteria:

  1. Interpretability: It can be understood using only the information the requester has seen (the "requester context").
  2. Language: It is written in the requester's language.
  3. Completeness: It actually answers the original request.

To measure this at scale, they developed a Reader Model (using Gemini 3 Flash). The model acts as a proxy for the user: it generates questions a confused reader might ask about specific spans of text (e.g., "What does 'run_42' refer to?") and then attempts to answer them using only the requester's context. If a question cannot be answered, an information gap is recorded.

They also tested two interventions to fix these gaps:

  • Explicit Requirements: Appending a message to the agent at the reporting stage, specifically instructing it to define new terms and answer the prompt first.
  • Secondary Rewriting: Using a separate model (DeepSeek V4 Pro) to take the agent's messy internal-facing reply and rewrite it for the user.

Datasets

The study evaluated 480 replies generated by eight different model families (including Claude Opus 5, GPT-5, DeepSeek, and Qwen). These agents worked on four distinct projects:

  • Research: Detecting mislabeled examples in datasets and long-document QA.
  • Business: Subscription pricing analysis and software onboarding redesign.

To validate the Reader Model, the authors created a gold set of 300 replies labeled by five human annotators with research backgrounds. The Reader Model achieved an 82% accuracy compared to the human majority, placing its performance within the range of individual human agreement.

Figure 2: (a) Cohen’s κ with the majority label of the gold set, which for an annotator is the majority of the other four. (b) Replies failed by the majority, by the number of gaps the reader finds.

Key Insights and Ideas

  • The Curse of Autonomy: The more "work" an agent does (i.e., the more tool-calling turns it takes), the more likely it is to fail at communicating. Failure rates jump from 40-48% for simple tasks to 70% for tasks requiring three or more turns.
  • Vocabulary Leakage: Agents don't just hallucinate; they "leak" their working vocabulary. Over 50% of the confusing labels found in failed replies originated in the agent's private files, project memory, or tool-call arguments.
  • Context Matters More than Personas: Simply telling a model "the user hasn't seen your work" (a notice) does almost nothing. However, giving the model a concrete checklist of what the reply must contain (requirements) significantly improves performance.
  • The "Aha" Moment: Usability is a distinct axis of performance. A model can be highly "intelligent" and successful at a task (like Claude Opus 5 in this study) but still have the highest failure rate in communication because it assumes the user shared its journey.

Key Results

  • Baseline Failure: Across all models, 54% of default replies failed usability. Claude Opus 5 had the highest failure rate (82%), while GPT-5.6 Sol had the lowest (30%).
  • Information Gaps: 46% of all replies contained at least one information gap. These gaps often appear early—in the first third of the message—stopping the reader before they get to the meat of the results.
  • Intervention Success:
    • Adding explicit requirements reduced the failure rate from 0.52 to 0.43.
  • Rewriting by a secondary model was even more effective, bringing the failure rate down to 0.36.
Figure 3: (a) Failure rate of each model with 95% intervals, divided into replies with an information gap and other failures. (b) Failure rate by tool-calling turns before the reply. (c) First occurrence of the coined labels in the agent’s context, where red marks the agent’s files, project memory and tool-call arguments.

Limitations and Future Work

The authors acknowledge several limitations:

  • Scripted Requesters: The interactions were synthetic and scripted rather than live human-in-the-loop sessions.
  • Homogeneous Readers: The "Reader Model" and human annotators represented a specific profile (knowledgeable in the field but unaware of the agent's process). Results might differ for laypeople or experts with different backgrounds.
  • Sampling: The sampling analysis was limited to one open-weight model.

Future work could involve applying these usability measures to real-world logged sessions and exploring how linguistic diversity is affected as LLMs increasingly shape the vocabulary of human-AI collaboration.

Summary updated

Abstract

People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.
BibTeX
@misc{jiang2026measuring,
  author = {Yilin Jiang and Fei Tan and Xuanyu Yin and Guanchu Wang and Xia Hu and Yutao Yue},
  title = {{Measuring and Reducing Unusable Replies of LLM Agents}},
  year = {2026},
  eprint = {2610.19659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.19659},
}
Figure