An Adaptive Framework for Generating Systematic Explanatory Answer in Online Q&A Platforms

Ziyang Chen, Xiaobin Wang, Yong Jiang, Jinzhi Liao, Pengjun Xie, Fei Huang, Xiang ZhaoarXiv 2024 · DOI · arXiv

Introduction

Traditional Retrieval-Augmented Generation (RAG) systems have revolutionized how Large Language Models (LLMs) access external knowledge. However, while these systems excel at "lookup" tasks—identifying specific facts like the capital of a country—they frequently falter when tasked with generating comprehensive, long-form explanatory answers. When faced with complex, multi-domain inquiries such as "Why has the price of gold increased recently?", standard RAG systems often produce disjointed responses that lack a cohesive logical thread.

The Limitations of Naive RAG

The authors identify two primary failures in "naive" RAG architectures that prevent them from matching human-level explanation quality:

  1. Information Fragmentation: Standard RAG retrieves isolated snippets. Without a macro-level structure, the final output often feels like a "bag of facts" rather than a reasoned argument.
  2. Shallow Coverage: Relying on simple keyword or embedding matches often causes the system to miss the "big picture." It fails to explore necessary sub-topics—such as historical context or international policy—that are essential for a truly comprehensive answer.
Comparison between the fragmented information retrieval of Naive RAG and the structured Adaptive Outlines of SynthRAG.

The SynthRAG Framework

To address these issues, the authors propose SynthRAG, a framework that moves away from a single-step "retrieve-then-generate" approach toward a three-stage "systematic thinking" process:

1. Adaptive Outline Generation

Instead of generating an answer immediately, the system first constructs a structured outline. By analyzing historical data from platforms like Zhihu, the system clusters questions into types and identifies "optimal" outline patterns found in high-quality human answers. This ensures the response follows a logical progression (e.g., History -> International Situation -> Supply & Demand).

2. Systematic Information Generation

Each section of the generated outline is treated as a specific sub-query. The system performs hierarchical retrieval for each subsection, generating detailed content in parallel. This dedicated focus on individual sub-topics ensures high information density and prevents the model from relying on general "hallucinations."

3. Customized Answer Generation

In the final stage, the model refines the integrated content using representative high-quality examples as a stylistic guide. This polishing step ensures the final output is a logically coherent essay rather than a mere collection of paragraphs.

Methodology and Data

The development of SynthRAG relied on several key datasets:

  • Zhihu-KOL: A dataset of 1 million real-world Q&A pairs used to learn optimal outline structures.
  • Reward Model Training Set: 100,000 answer preference pairs based on user upvotes, used to train a model to distinguish between high-quality and low-quality responses.
  • Evaluation Set: 50 "hot" questions from early 2024 were used to test the model's ability to handle current, complex social dynamics.

Key Results and Impact

The empirical results demonstrate that SynthRAG significantly outperforms existing benchmarks in long-form QA:

  • Superiority over GPT-4: In evaluations using GPT-4 as a judge, SynthRAG achieved an overall score of 4.52/5, surpassing both vanilla GPT-4 (3.82) and standard RAG (4.12).
  • Information Richness: Using a factual QA proxy test, SynthRAG achieved an information score of 85.76%, compared to 66.43% for standard RAG.
  • Real-World Deployment: In a live deployment on the Zhihu platform, SynthRAG answers outperformed 79.8% of human contributors and ranked in the top 2% of all upvoted responses.

Conclusion and Future Work

While SynthRAG represents a major step forward in systematic generation, it faces challenges regarding computational latency, currently taking 2 to 3 minutes to generate a single answer. Future work will focus on optimizing this speed and introducing personalization features to tailor the depth and tone of explanations to specific user expertise levels.

Summary updated

Abstract

Question Answering (QA) systems face challenges in handling complex questions that require multi-domain knowledge synthesis. The naive RAG models, although effective in information retrieval, struggle with complex questions that require comprehensive and in-depth answers. The pioneering task is defined as explanatory answer generation, which entails handling identified challenges such as the requirement for comprehensive information and logical coherence within the generated context. To address these issues, we refer to systematic thinking theory and propose SynthRAG, an innovative framework designed to enhance QA performance. SynthRAG improves on conventional models by employing adaptive outlines for dynamic content structuring, generating systematic information to ensure detailed coverage, and producing customized answers tailored to specific user inquiries. This structured approach guarantees logical coherence and thorough integration of information, yielding responses that are both insightful and methodically organized. Empirical evaluations underscore SynthRAG's effectiveness, demonstrating its superiority in handling complex questions, overcoming the limitations of naive RAG models, and significantly improving answer quality and depth. Furthermore, an online deployment on the Zhihu platform revealed that SynthRAG's answers achieved notable user engagement, with each response averaging 5.73 upvotes and surpassing the performance of 79.8% of human contributors, highlighting the practical relevance and impact of the proposed framework. Our code is available at https://github.com/czy1999/SynthRAG .
BibTeX
@misc{chen2024adaptive,
  author = {Ziyang Chen and Xiaobin Wang and Yong Jiang and Jinzhi Liao and Pengjun Xie and Fei Huang and Xiang Zhao},
  title = {{An Adaptive Framework for Generating Systematic Explanatory Answer in Online Q\&A Platforms}},
  year = {2024},
  eprint = {2410.17694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.17694},
}
Figure