Abstract: Justin Xu, Yiming Li, Zizheng Zhang, Augustine Yui Hei Luk, Mayank Jobanputra, Samarth Oza, Ashley Murray, Meghana Reddy Kasula, Andrew Parker, David W Eyre. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
2025.
Introduction
Large Language Models (LLMs) often struggle with hallucination and poor attribution, particularly in complex multi-hop reasoning tasks where errors can cascade. In high-stakes domains like medicine, the "black box" nature of these models is a significant barrier to adoption. To address this, researchers have developed Tree-of-Quote (ToQ) prompting, a framework that integrates internal quoting as a fundamental part of the reasoning process rather than a post-hoc addition.
The Tree-of-Quote (ToQ) Framework
ToQ structures reasoning as a dynamic graph (implemented via LangGraph), ensuring that every step in a chain of thought is grounded in verifiable evidence. The process is managed through four primary nodes:
- Initialization: The model decomposes a complex question into simpler sub-questions.
- Quoting: The model generates internal quotes from its training data (e.g., "According to PubMed...") to answer sub-questions without external retrieval.
- Scoring: Quotes are evaluated using the QUIP (Quoted Information Precision) metric. If a quote falls below a quality threshold, the system triggers a retry or prunes that reasoning branch.
- Questioning: The model assesses if it has sufficient context for a final answer or if further sub-questions are required.
Evaluation and Datasets
The framework was evaluated across five diverse datasets, including general multi-hop QA (StrategyQA, 2WikiMultiHopQA, MuSiQue, MoreHopQA) and medical reasoning (MedQA). To validate the results, the researchers introduced the FQ-Score (Factuality with Quoting Score), a holistic metric combining correctness, attribution quality, and logical flow. This metric was validated through reader studies involving clinicians to ensure alignment with expert human judgment.
Key Results and Insights
ToQ consistently outperforms standard prompting techniques like Chain-of-Thought (CoT) and Chain-of-Verification (CoVe). On StrategyQA, ToQ achieved 81.7% accuracy with GPT-4o. In medical contexts, it provided a transparent "paper trail," with human evaluators confirming the accuracy of 37 out of 45 generated quotes.
A core insight of ToQ is that the act of quoting serves as a factual filter, catching hallucinations early in the reasoning chain. The study also demonstrates that LLMs often possess the necessary facts within their weights; they simply require structured prompting to access them reliably.
Limitations and Future Directions
While effective, ToQ is computationally more expensive than standard prompting, requiring roughly 2x more LLM calls and tokens due to its iterative loops. Future work aims to integrate ToQ with external Retrieval-Augmented Generation (RAG) systems and optimize the framework for smaller, more efficient language models to reduce operational costs.
Summary updated