AcademicGPT: Empowering Academic Research

Shufa Wei, Xiaolong Xu, Xianbiao Qi, Xi Yin, Jun Xia, Jingyi Ren, Peijun Tang, Yuxiang Zhong, Yihao Chen, Xiaoqin Ren, Yuxin Liang, Liankai Huang, Kai Xie, Weikang Gui, Wei Tan, Shuanglong Sun, Yongquan Hu, Qinxian Liu, Nanjin Li, Chihao Dai, Lihua Wang, Xiaohui Liu, Lei Zhang, Yutao XiearXiv 2023 · DOI · OpenReview

TL;DR

This paper introduces AcademicGPT, a specialized Large Language Model (LLM) built upon LLaMA2-70B, fine-tuned on a massive academic corpus to excel in research-related tasks. It demonstrates strong performance across general and academic benchmarks, and powers several novel applications like AI-assisted paper review and question answering, aiming to significantly boost academic productivity.

What Problem Does This Paper Address?

The paper addresses the challenge that while general-purpose LLMs are powerful, they often lack the specialized knowledge and nuanced understanding required for academic research. Researchers face an overwhelming volume of literature, making it difficult to stay abreast of new findings. Existing LLMs, trained on broad datasets, may not perform optimally on domain-specific tasks like analyzing academic papers, answering complex research questions, or generating scholarly content. AcademicGPT aims to bridge this gap by creating an LLM specifically tailored for the academic domain, enhancing efficiency and depth in research activities.

Methods

AcademicGPT is a continual pre-training of the LLaMA2-70B model. The core methodology involves:

  • Continual Pre-training: The LLaMA2-70B model is further trained on a curated dataset comprising academic papers, theses, and high-quality Chinese data. This process aims to imbue the model with specialized academic knowledge and improve its Chinese language capabilities.
  • Domain-Specific Fine-tuning: While the paper focuses on continual pre-training, the resulting AcademicGPT model is then used as a foundation for various applications, implying further fine-tuning for specific tasks.
  • ReAct Framework for Q&A: For the General Academic Question Answering application, the system leverages the ReAct framework, which combines reasoning and action, to enable multi-turn dialogues, tool usage (like search engines and knowledge graphs), and context-aware responses.
  • Two-Stage Pipeline for Paper Reading: AI-assisted paper reading is achieved through a two-stage process: first, continued pre-training with a dynamic ntk-aware method to extend context windows, followed by supervised fine-tuning on a dataset like LongAlpaca-12k.

Datasets

  • Training Data:
    • Academic papers, theses, and content from academic domains.
  • High-quality Chinese data (Common Crawl, Wiki, Baike, Books).
  • Specific academic sources include: Top Universities theses, Arxiv, Unpaywall, PubMed, Semantic Scholar, Falcon Refinedweb, and Wiki (English).
  • The paper mentions a total of 370.85 billion tokens for continual pre-training.
  • Evaluation Benchmarks:
    • General: MMLU, CEval.
  • Academic/Specialized: PubMedQA, SCIEval.
  • Newly Created: ComputerScienceQA (designed to evaluate models on computer science knowledge).
  • Application-Specific Datasets:
    • LongAlpaca-12k (for AI-assisted paper reading).
  • OpenReview data (for Paper Review).
  • Extensive corpus of 1 million research papers (for Title and Abstract Generation).

Key Insights and Ideas

  • Domain Specialization is Key: The research strongly suggests that while general LLMs are versatile, fine-tuning on domain-specific corpora like academic literature significantly enhances their performance on specialized tasks.
  • Continual Pre-training Enhances Capabilities: AcademicGPT's performance improvements on academic benchmarks, without significant degradation on general ones, highlight the effectiveness of continual pre-training for domain adaptation.
  • LLMs as Research Assistants: The development of applications like AI-assisted paper reading, review, and question answering demonstrates the potential of LLMs to act as powerful tools for researchers, streamlining workflows and accelerating discovery.
  • ReAct Framework for Complex Q&A: The integration of the ReAct framework allows for more sophisticated and contextually aware academic question answering, mimicking human reasoning processes.
  • Bridging the Gap in Paper Review: The paper shows promising results in AI-assisted paper review, suggesting LLMs can contribute to the peer-review process, though a gap with human reviewers still exists.

Key Results

  • MMLU: AcademicGPT achieved an accuracy of 0.688, outperforming LLaMA1 (0.634) and ChatGPT (0.664), and slightly behind LLaMA2-70B (0.693). Performance improved on categories well-represented in the training data.
  • CEval (Chinese): AcademicGPT achieved 0.551, an improvement from LLaMA2-70B's 0.508, demonstrating enhanced Chinese language capabilities.
  • PubMedQA: AcademicGPT achieved 0.806, outperforming LLaMA1, LLaMA2, ChatGPT3.5, and GPT4, likely due to the inclusion of medical-related corpus.
  • SCIEval: AcademicGPT achieved an average accuracy of 0.688, surpassing LLaMA2 (0.636) and ChatGPT (0.679), particularly strong in physics.
  • ComputerScienceQA: AcademicGPT achieved an overall accuracy of 0.835, significantly better than LLaMA2-70B (0.799) and ChatGPT (0.811).
  • Paper Review: AcademicGPT achieved a final recommendation accuracy of 68.4%, showing capability in this complex task.

Limitations and Future Work

  • Data Scale: While extensive, the authors acknowledge that the AcademicGPT training corpus might not be as large as some general-purpose LLMs.
  • Performance Nuances: Performance on certain categories in MMLU slightly decreased if those categories were not heavily covered in the training data.
  • Paper Review Gap: While capable, AcademicGPT still shows a noticeable gap compared to human reviewers in paper review tasks.
  • Future Work: The paper implicitly suggests future work in expanding the training data, further refining the model for specific academic sub-domains, and enhancing the capabilities of the AI-powered research tools. The development of domain-specific LLMs is seen as a transformative step for academic research.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Yet, many of these advanced LLMs are tailored for broad, general-purpose applications. In this technical report, we introduce AcademicGPT, designed specifically to empower academic research. AcademicGPT is a continual training model derived from LLaMA2-70B. Our training corpus mainly consists of academic papers, thesis, content from some academic domain, high-quality Chinese data and others. While it may not be extensive in data scale, AcademicGPT marks our initial venture into a domain-specific GPT tailored for research area. We evaluate AcademicGPT on several established public benchmarks such as MMLU and CEval, as well as on some specialized academic benchmarks like PubMedQA, SCIEval, and our newly-created ComputerScienceQA, to demonstrate its ability from general knowledge ability, to Chinese ability, and to academic ability. Building upon AcademicGPT's foundation model, we also developed several applications catered to the academic area, including General Academic Question Answering, AI-assisted Paper Reading, Paper Review, and AI-assisted Title and Abstract Generation.
BibTeX
@misc{wei2023academicgpt,
  author = {Shufa Wei and Xiaolong Xu and Xianbiao Qi and Xi Yin and Jun Xia and Jingyi Ren and Peijun Tang and Yuxiang Zhong and Yihao Chen and Xiaoqin Ren and Yuxin Liang and Liankai Huang and Kai Xie and Weikang Gui and Wei Tan and Shuanglong Sun and Yongquan Hu and Qinxian Liu and Nanjin Li and Chihao Dai and Lihua Wang and Xiaohui Liu and Lei Zhang and Yutao Xie},
  title = {{AcademicGPT: Empowering Academic Research}},
  year = {2023},
  eprint = {2311.12315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.12315},
}
Figure