SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model

Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Qian Li, Bo Wang, Shijin Wang, Zhixiong Zhang, Guoping Hu

Abstract

Dayong Wu, Jiaqi Li, Baoxin Wang, Honghong Zhao, Siyuan Xue, Yanjie Yang, Zhijun Chang, Rui Zhang, Li Qian, Bo Wang, Shijin Wang, Zhixiong Zhang, Guoping Hu. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations.

2024.

Introduction

The rapid expansion of scientific literature presents a significant challenge for researchers attempting to stay current with their fields. While general-purpose Large Language Models (LLMs) like GPT-4 have shown promise in assisting with text-based tasks, they often struggle with the precision, nomenclature, and structural complexity inherent in academic papers. SparkRA is a specialized knowledge service system designed to address these gaps. Built upon a custom-tuned model called SciLit-LLM, SparkRA integrates Retrieval-Augmented Generation (RAG) to provide a robust toolset for literature investigation, paper reading, and academic writing.

The Challenge of Scientific Literature

General-purpose LLMs face three primary hurdles in the scientific domain:

  1. Hallucinations: Models may fabricate citations or facts when domain-specific data is missing from their training set.
  2. Context Limitations: The length and technical density of academic papers often exceed the effective context windows of standard models, leading to loss of detail.
  3. Specialized Tasks: Academic polishing and translation require a deep mastery of formal scientific styles and domain-specific terminology that general models often lack.

System Architecture and SciLit-LLM

The core of SparkRA is SciLit-LLM, a 13B parameter model derived from the iFLYTEK Spark LLM. The development of SciLit-LLM involved a multi-stage process starting with domain-specific continual pre-training on a corpus of over 10 million academic papers and patents. This was followed by Supervised Fine-Tuning (SFT) using a "dual approach" that combined AI-generated instructions (Self-instruct) with expert-curated human examples.

The detailed system architecture of SparkRA, illustrating the integration of the Literature DB, SciLit-LLM, and specialized modules for research tasks.

To ensure accuracy, SparkRA employs a Retrieval-Augmented Generation (RAG) framework. When a user submits a query, the system retrieves relevant snippets from its massive internal database to ground the model's responses in verified literature. This approach allows the system to "zoom in" on relevant sections of long papers without the computational overhead of massive context windows.

Specialized Research Modules

SparkRA is organized into three primary functional areas:

  • Literature Investigation: Automatically analyzes and summarizes multiple papers to generate research reviews. It utilizes query rewriting to transform informal user questions into optimized search terms.
  • Paper Reading: Acts as a "copilot" for deep reading, using chapter recognition and text splitting to maintain semantic integrity while answering specific questions about a document.
  • Academic Writing: Provides one-click translation and polishing. It uses "Dynamic Perception Prompts" to help the model recognize and correctly handle domain-specific terminology, ensuring high fidelity to the original meaning.

Performance Evaluation

SparkRA was evaluated against leading models including GPT-4, GPT-3.5, and Llama3-8B using Mean Opinion Scores (MOS) from human experts. The results demonstrate that domain specialization can allow a smaller model (13B) to outperform much larger general-purpose counterparts in niche tasks.

Comparison of SparkRA against other LLMs in paper reading tasks, showing superior scores in Factuality and Informativeness.

In paper reading evaluations, SparkRA achieved a Factuality score of 4.68/5, narrowly surpassing GPT-4 (4.67). In academic polishing, it led in both Academic Quality (4.61) and Fidelity (4.45). Furthermore, the model demonstrated a higher "rejection tendency," meaning it was more likely to admit when information was unavailable rather than hallucinating a false answer—a critical feature for maintaining scientific integrity.

Conclusion and Future Work

As of July 2024, SparkRA has attracted over 50,000 registered users and processed over 1.3 million requests, proving its utility in the research community. While the system currently excels in fidelity and factuality, future work will focus on scaling the "Review Generation" feature beyond its current 30-paper limit and further improving the fluency of its academic translations to match the natural prose of the largest general models.

Summary updated

BibTeX
@misc{wu2024sparkra,
  author = {Dayong Wu and Jiaqi Li and Baoxin Wang and Honghong Zhao and Siyuan Xue and Yanjie Yang and Zhijun Chang and Rui Zhang and Qian Li and Bo Wang and Shijin Wang and Zhixiong Zhang and Guoping Hu},
  title = {{SparkRA: A Retrieval-Augmented Knowledge Service System Based on Spark Large Language Model}},
  year = {2024},
  eprint = {2408.06574},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.06574},
}
Figure