TL;DR
Researchers have introduced CryptoAnalystBench, a new benchmark designed to evaluate how well LLM "analyst agents" handle complex, long-form reasoning using multiple tools. By testing frontier models on real-world cryptocurrency queries, the study reveals that while models are getting better at avoiding blatant hallucinations, they frequently fail at higher-order tasks like reconciling conflicting data sources or maintaining temporal awareness.
What Problem Does This Paper Address?
We are rapidly moving from LLMs as simple chatbots to "analyst agents" that inform high-stakes decisions in finance, law, and medicine. These agents don't just "know" things; they must orchestrate dozens of tool calls (APIs, web searches, database queries) and synthesize hundreds of tokens of evidence into a coherent report.
Existing benchmarks usually focus on either simple tool-calling (did the model call the right API?) or basic factuality (is the answer true?). However, there is a massive gap in evaluating the intersection: how do models perform when they must integrate large volumes of dynamic, often contradictory, multi-tool outputs? The authors use the cryptocurrency/DeFi domain as a "stress test" because it is characterized by extreme data density, high volatility (prices change by the minute), and significant financial stakes.
Methods
The authors developed a three-part framework to study these agentic failures:
- The Agentic Harness: A ReAct-style reasoning loop equipped with production-grade tools, including market data APIs (CoinGecko, DefiLlama), web search, and code execution environments.
- Multi-Layered Evaluation:
- Automated Citation Verification: A pipeline that extracts atomic claims and maps them back to specific tool outputs to check for grounding.
- LLM-as-a-Judge: Using a structured rubric (scored 1–10) across four dimensions: Temporal Relevance, Data Consistency, Depth, and Relevance.
- Error Taxonomy: Through human annotation, the researchers identified seven higher-order failure modes that standard metrics often miss, such as "Source Reconciliation Failure" (failing to acknowledge when two different tools provide different prices for the same token).
Datasets
The primary contribution is CryptoAnalystBench, which consists of:
- 198 production-aligned queries curated from real-world user traffic.
- 11 categories including Risk Assessment, Protocol Comparison, and Yield Optimization.
- Unlike static benchmarks, these queries are "time-robust," meaning the question remains valid even as the answer changes with market conditions.
Key Insights and Ideas
- The "Hallucination" Shift: Blatant fabrication is becoming rare (less than 6% across models). The real danger has shifted to structural failures—models might cite real data but fail to realize that data is 24 hours out of date or contradicts a previous sentence in the same report.
- The Seven Sins of Analyst Agents: The paper defines a taxonomy of failures:
- Staleness: Using outdated info despite having access to fresh APIs.
- Inconsistent Claims: Internal contradictions.
- Source Reconciliation Failure: Ignoring conflicts between different tools.
- Shallow Synthesis: Listing facts without meaningful analysis.
- Missing Risk Context: Omitting crucial "fine print" (e.g., undisclosed promotions).
- Overconfident Prediction: Making hard claims without uncertainty bounds.
- Partial/Misframed Answer: Only answering half the prompt.
- Judge vs. Human: While LLM judges (like DeepSeek V3.1) only show "fair-to-moderate" agreement with humans on precise scalar scores, they are surprisingly reliable at identifying specific failure modes, making them useful for scalable developer feedback.
Key Results
The study evaluated several frontier models, including GPT-5.2, Kimi k2.5, and Qwen 235B.
- Performance Saturation: Most models scored high (>9.0) on basic relevance, but struggled significantly with Depth and Data Consistency.
- Model Profiles: GPT-5.2 (as named in the paper) showed the highest consistency, while Kimi k2.5 excelled in depth and technical detail.
- Staleness is Persistent: Even the best models exhibited "Staleness" in 4.5% to 17.2% of their responses, often preferring a cached web search result over a fresh API data point.
- Mitigation Success: The authors tested "targeted mitigations"—like forcing the model to prioritize API outputs over search results. This improved performance for high-capability models but actually "overburdened" weaker models, causing them to regress.
Limitations and Future Work
- Subjectivity: Despite the rubric, "Depth" and "Relevance" remain somewhat subjective, and human-model alignment on exact scores is still a work in progress.
- Domain Specificity: While the crypto domain is a great proxy for high-density data, the findings need to be validated in other fields like legal or medical analysis.
- Future Directions: The authors suggest moving toward "adaptive mitigation" strategies, where the agent changes its reasoning behavior based on the complexity of the task or the reliability of the tools it is currently using.
Summary updated