Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence

Junjie Wang, Yuxiang Zhang, Lin Zhang, Ping Yang, Xinyu Gao, Ziwei Wu, Xiaoqun Dong, Junqing He, Jianheng Zhuo, Qi Yang, Yongfeng Huang, Xiayu Li, Yanghan Wu, Junyu Lu, Xinyu Zhu, Weifeng Chen, Ting Han, Kunhao Pan, Rui Wang, Hao Wang, Xiaojun Wu, Zhongshen Zeng, Chongpei Chen, Ruyi Gan, Jiaxing ZhangarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces Fengshenbang, an ambitious open-source project aiming to build a comprehensive foundation for Chinese cognitive intelligence. It addresses the critical gap in Chinese-language foundation models by providing a suite of pre-trained models, a user-friendly framework, and robust benchmarks, all designed to empower the Chinese NLP community.

What Problem Does This Paper Address?

The rapid advancement of foundation models, particularly large language models (LLMs), has revolutionized AI, enabling general intelligence and tackling complex tasks. However, a significant challenge exists: the overwhelming dominance of English-language models and resources. This leaves the Chinese language community underserved, lacking systematic research support and accessible tools. Furthermore, many existing foundation models are resource-intensive, posing a barrier for users with limited computational power. Fengshenbang aims to bridge this gap by creating a dedicated, open-source ecosystem for Chinese NLP, democratizing access to powerful models and fostering community-driven development.

Methods

Fengshenbang is structured into three core sub-projects:

  • Fengshenbang Model: This component offers a diverse collection of pre-trained models covering various NLP tasks, including Natural Language Understanding (NLU), Natural Language Generation (NLG), and Natural Language Transformation (NLT). The models are categorized using a "User-centered Taxonomy" (UCT) to align with user needs and are named according to a clear convention based on Chinese mythology. The project emphasizes diversity in model architectures and sizes to cater to different resource constraints.
  • Fengshen Framework: Designed for user-friendliness and efficiency, this framework integrates popular tools like Huggingface, Megatron-LM, and PyTorch-Lightning. It simplifies distributed training, memory optimization, and fine-tuning for large-scale models, even those exceeding 10 billion parameters. The framework provides standardized data processing, tutorials, and APIs to facilitate model implementation and innovation.
  • Fengshen Benchmark: To ensure fair evaluation and track progress in the Chinese NLP landscape, Fengshenbang introduces a suite of benchmarks. These benchmarks are designed to be linguistically distinct from English counterparts and aim to be future-oriented, considering broader ethical, technical, and societal challenges. They also strive for applicability by reflecting real-world scenarios.

Datasets

The paper mentions the use of the WuDao Corpora (180 GB version) for pre-training various models. For specific models, datasets like CLURCorpus2020 and PubMed abstracts are also utilized. The project also aims to release a general pre-training dataset and is developing benchmarks like Chinese-SuperGLUE and QAKM (Question Answering with Knowledge Models).

Key Insights and Ideas

  • Democratizing Chinese NLP: The core idea is to create a comprehensive, open-source ecosystem specifically for Chinese language AI, addressing a significant gap in the current landscape.
  • User-Centric Design: The project prioritizes user needs through a taxonomy that maps requirements to models and a framework that simplifies access and utilization, even for those with limited resources.
  • Mythology-Inspired Naming: The use of names from the Chinese novel "Fengshen Bang" for model series adds a unique cultural touch and provides a memorable organizational structure.
  • Bridging the Monolingual Divide: The development of Chinese-specific benchmarks like Chinese-SuperGLUE is crucial for fair evaluation and to prevent research from being solely English-centric.
  • Community-Driven Development: Fengshenbang is positioned as an ongoing, collaborative effort, inviting contributions from individuals and organizations to foster a thriving Chinese NLP ecosystem.

Key Results

While the paper focuses on introducing the project and its components, it highlights some achievements:

  • Top Performance on Benchmarks: The Erlangshen-MegatronBert model has achieved top rankings on several Chinese NLP benchmarks, including FewCLUE, ZeroCLUE, and CLUE's semantic matching task.
  • Large-Scale Models: The project has successfully pre-trained large models, including Erlangshen-MegatronBert-1.3B, which was the largest BERT in the Chinese open-source community at its release.
  • Framework Efficiency: The Fengshen Framework claims to offer superior performance compared to standard PyTorch, with up to 300% acceleration in training.
  • Extensive Model Availability: The project lists 49 models available through Huggingface, covering a wide range of tasks and architectures.

Limitations and Future Work

  • Ongoing Development: Fengshenbang is presented as a continuously evolving project, with sub-projects being updated regularly.
  • Ethical Considerations: The authors acknowledge the potential ethical implications of large-scale models, including bias, environmental impact, and misuse, and plan to provide a more detailed study in future reports.
  • Exploration and Collaboration: The project actively seeks collaboration with other organizations for experimental explorations and the development of new models.
  • Future Benchmarks: The paper mentions plans to release a general pre-training dataset and further develop benchmarks to address broader ethical, technical, and societal challenges.
  • UnifiedMC and MAP: Some models, like UnifiedMC and MAP, are marked as temporary, indicating ongoing research and development.

Summary updated

Abstract

Nowadays, foundation models become one of fundamental infrastructures in artificial intelligence, paving ways to the general intelligence. However, the reality presents two urgent challenges: existing foundation models are dominated by the English-language community; users are often given limited resources and thus cannot always use foundation models. To support the development of the Chinese-language community, we introduce an open-source project, called Fengshenbang, which leads by the research center for Cognitive Computing and Natural Language (CCNL). Our project has comprehensive capabilities, including large pre-trained models, user-friendly APIs, benchmarks, datasets, and others. We wrap all these in three sub-projects: the Fengshenbang Model, the Fengshen Framework, and the Fengshen Benchmark. An open-source roadmap, Fengshenbang, aims to re-evaluate the open-source community of Chinese pre-trained large-scale models, prompting the development of the entire Chinese large-scale model community. We also want to build a user-centered open-source ecosystem to allow individuals to access the desired models to match their computing resources. Furthermore, we invite companies, colleges, and research institutions to collaborate with us to build the large-scale open-source model-based ecosystem. We hope that this project will be the foundation of Chinese cognitive intelligence.
BibTeX
@misc{wang2022fengshenbang,
  author = {Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen and Ruyi Gan and Jiaxing Zhang},
  title = {{Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence}},
  year = {2022},
  eprint = {2209.02970},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.02970},
}
Figure