CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis

Anushri Eswaran, Oleg Golev, Darshan Tank, Sidhant Rahi, Himanshu Tyagi

TL;DR

Researchers have introduced CryptoAnalystBench, a new benchmark designed to evaluate how well LLM "analyst agents" handle complex, long-form reasoning using multiple tools. By testing frontier models on real-world cryptocurrency queries, the study reveals that while models are getting better at avoiding blatant hallucinations, they frequently fail at higher-order tasks like reconciling conflicting data sources or maintaining temporal awareness.

What Problem Does This Paper Address?

We are rapidly moving from LLMs as simple chatbots to "analyst agents" that inform high-stakes decisions in finance, law, and medicine. These agents don't just "know" things; they must orchestrate dozens of tool calls (APIs, web searches, database queries) and synthesize hundreds of tokens of evidence into a coherent report.

Existing benchmarks usually focus on either simple tool-calling (did the model call the right API?) or basic factuality (is the answer true?). However, there is a massive gap in evaluating the intersection: how do models perform when they must integrate large volumes of dynamic, often contradictory, multi-tool outputs? The authors use the cryptocurrency/DeFi domain as a "stress test" because it is characterized by extreme data density, high volatility (prices change by the minute), and significant financial stakes.

Methods

The authors developed a three-part framework to study these agentic failures:

  1. The Agentic Harness: A ReAct-style reasoning loop equipped with production-grade tools, including market data APIs (CoinGecko, DefiLlama), web search, and code execution environments.
  2. Multi-Layered Evaluation:
    • Automated Citation Verification: A pipeline that extracts atomic claims and maps them back to specific tool outputs to check for grounding.
  3. LLM-as-a-Judge: Using a structured rubric (scored 1–10) across four dimensions: Temporal Relevance, Data Consistency, Depth, and Relevance.
  4. Error Taxonomy: Through human annotation, the researchers identified seven higher-order failure modes that standard metrics often miss, such as "Source Reconciliation Failure" (failing to acknowledge when two different tools provide different prices for the same token).

Datasets

The primary contribution is CryptoAnalystBench, which consists of:

  • 198 production-aligned queries curated from real-world user traffic.
  • 11 categories including Risk Assessment, Protocol Comparison, and Yield Optimization.
  • Unlike static benchmarks, these queries are "time-robust," meaning the question remains valid even as the answer changes with market conditions.

Key Insights and Ideas

  • The "Hallucination" Shift: Blatant fabrication is becoming rare (less than 6% across models). The real danger has shifted to structural failures—models might cite real data but fail to realize that data is 24 hours out of date or contradicts a previous sentence in the same report.
  • The Seven Sins of Analyst Agents: The paper defines a taxonomy of failures:
  • Staleness: Using outdated info despite having access to fresh APIs.
  • Inconsistent Claims: Internal contradictions.
  • Source Reconciliation Failure: Ignoring conflicts between different tools.
  • Shallow Synthesis: Listing facts without meaningful analysis.
  • Missing Risk Context: Omitting crucial "fine print" (e.g., undisclosed promotions).
  • Overconfident Prediction: Making hard claims without uncertainty bounds.
  • Partial/Misframed Answer: Only answering half the prompt.
  • Judge vs. Human: While LLM judges (like DeepSeek V3.1) only show "fair-to-moderate" agreement with humans on precise scalar scores, they are surprisingly reliable at identifying specific failure modes, making them useful for scalable developer feedback.

Key Results

The study evaluated several frontier models, including GPT-5.2, Kimi k2.5, and Qwen 235B.

  • Performance Saturation: Most models scored high (>9.0) on basic relevance, but struggled significantly with Depth and Data Consistency.
  • Model Profiles: GPT-5.2 (as named in the paper) showed the highest consistency, while Kimi k2.5 excelled in depth and technical detail.
  • Staleness is Persistent: Even the best models exhibited "Staleness" in 4.5% to 17.2% of their responses, often preferring a cached web search result over a fresh API data point.
  • Mitigation Success: The authors tested "targeted mitigations"—like forcing the model to prioritize API outputs over search results. This improved performance for high-capability models but actually "overburdened" weaker models, causing them to regress.

Limitations and Future Work

  • Subjectivity: Despite the rubric, "Depth" and "Relevance" remain somewhat subjective, and human-model alignment on exact scores is still a work in progress.
  • Domain Specificity: While the crypto domain is a great proxy for high-density data, the findings need to be validated in other fields like legal or medical analysis.
  • Future Directions: The authors suggest moving toward "adaptive mitigation" strategies, where the agent changes its reasoning behavior based on the complexity of the task or the reliability of the tools it is currently using.

Summary updated

Abstract

Modern analyst agents must reason over complex, high token inputs, including dozens of retrieved documents, tool outputs, and time sensitive data. While prior work has produced tool calling benchmarks and examined factuality in knowledge augmented systems, relatively little work studies their intersection: settings where LLMs must integrate large volumes of dynamic, structured and unstructured multi tool outputs. We investigate LLM failure modes in this regime using crypto as a representative high data density domain. We introduce (1) CryptoAnalystBench, an analyst aligned benchmark of 198 production crypto and DeFi queries spanning 11 categories; (2) an agentic harness equipped with relevant crypto and DeFi tools to generate responses across multiple frontier LLMs; and (3) an evaluation pipeline with citation verification and an LLM as a judge rubric spanning four user defined success dimensions: relevance, temporal relevance, depth, and data consistency. Using human annotation, we develop a taxonomy of seven higher order error types that are not reliably captured by factuality checks or LLM based quality scoring. We find that these failures persist even in state of the art systems and can compromise high stakes decisions. Based on this taxonomy, we refine the judge rubric to better capture these errors. While the judge does not align with human annotators on precise scoring across rubric iterations, it reliably identifies critical failure modes, enabling scalable feedback for developers and researchers studying analyst style agents. We release CryptoAnalystBench with annotated queries, the evaluation pipeline, judge rubrics, and the error taxonomy, and outline mitigation strategies and open challenges in evaluating long form, multi tool augmented systems.
BibTeX
@misc{eswaran2026cryptoanalystbench,
  author = {Anushri Eswaran and Oleg Golev and Darshan Tank and Sidhant Rahi and Himanshu Tyagi},
  title = {{CryptoAnalystBench: Failures in Multi-Tool Long-Form LLM Analysis}},
  year = {2026},
  eprint = {2602.11304},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.11304},
}
Figure