TL;DR
This paper introduces Fengshenbang, an ambitious open-source project aiming to build a comprehensive foundation for Chinese cognitive intelligence. It addresses the critical gap in Chinese-language foundation models by providing a suite of pre-trained models, a user-friendly framework, and robust benchmarks, all designed to empower the Chinese NLP community.
What Problem Does This Paper Address?
The rapid advancement of foundation models, particularly large language models (LLMs), has revolutionized AI, enabling general intelligence and tackling complex tasks. However, a significant challenge exists: the overwhelming dominance of English-language models and resources. This leaves the Chinese language community underserved, lacking systematic research support and accessible tools. Furthermore, many existing foundation models are resource-intensive, posing a barrier for users with limited computational power. Fengshenbang aims to bridge this gap by creating a dedicated, open-source ecosystem for Chinese NLP, democratizing access to powerful models and fostering community-driven development.
Methods
Fengshenbang is structured into three core sub-projects:
- Fengshenbang Model: This component offers a diverse collection of pre-trained models covering various NLP tasks, including Natural Language Understanding (NLU), Natural Language Generation (NLG), and Natural Language Transformation (NLT). The models are categorized using a "User-centered Taxonomy" (UCT) to align with user needs and are named according to a clear convention based on Chinese mythology. The project emphasizes diversity in model architectures and sizes to cater to different resource constraints.
- Fengshen Framework: Designed for user-friendliness and efficiency, this framework integrates popular tools like Huggingface, Megatron-LM, and PyTorch-Lightning. It simplifies distributed training, memory optimization, and fine-tuning for large-scale models, even those exceeding 10 billion parameters. The framework provides standardized data processing, tutorials, and APIs to facilitate model implementation and innovation.
- Fengshen Benchmark: To ensure fair evaluation and track progress in the Chinese NLP landscape, Fengshenbang introduces a suite of benchmarks. These benchmarks are designed to be linguistically distinct from English counterparts and aim to be future-oriented, considering broader ethical, technical, and societal challenges. They also strive for applicability by reflecting real-world scenarios.
Datasets
The paper mentions the use of the WuDao Corpora (180 GB version) for pre-training various models. For specific models, datasets like CLURCorpus2020 and PubMed abstracts are also utilized. The project also aims to release a general pre-training dataset and is developing benchmarks like Chinese-SuperGLUE and QAKM (Question Answering with Knowledge Models).
Key Insights and Ideas
- Democratizing Chinese NLP: The core idea is to create a comprehensive, open-source ecosystem specifically for Chinese language AI, addressing a significant gap in the current landscape.
- User-Centric Design: The project prioritizes user needs through a taxonomy that maps requirements to models and a framework that simplifies access and utilization, even for those with limited resources.
- Mythology-Inspired Naming: The use of names from the Chinese novel "Fengshen Bang" for model series adds a unique cultural touch and provides a memorable organizational structure.
- Bridging the Monolingual Divide: The development of Chinese-specific benchmarks like Chinese-SuperGLUE is crucial for fair evaluation and to prevent research from being solely English-centric.
- Community-Driven Development: Fengshenbang is positioned as an ongoing, collaborative effort, inviting contributions from individuals and organizations to foster a thriving Chinese NLP ecosystem.
Key Results
While the paper focuses on introducing the project and its components, it highlights some achievements:
- Top Performance on Benchmarks: The Erlangshen-MegatronBert model has achieved top rankings on several Chinese NLP benchmarks, including FewCLUE, ZeroCLUE, and CLUE's semantic matching task.
- Large-Scale Models: The project has successfully pre-trained large models, including Erlangshen-MegatronBert-1.3B, which was the largest BERT in the Chinese open-source community at its release.
- Framework Efficiency: The Fengshen Framework claims to offer superior performance compared to standard PyTorch, with up to 300% acceleration in training.
- Extensive Model Availability: The project lists 49 models available through Huggingface, covering a wide range of tasks and architectures.
Limitations and Future Work
- Ongoing Development: Fengshenbang is presented as a continuously evolving project, with sub-projects being updated regularly.
- Ethical Considerations: The authors acknowledge the potential ethical implications of large-scale models, including bias, environmental impact, and misuse, and plan to provide a more detailed study in future reports.
- Exploration and Collaboration: The project actively seeks collaboration with other organizations for experimental explorations and the development of new models.
- Future Benchmarks: The paper mentions plans to release a general pre-training dataset and further develop benchmarks to address broader ethical, technical, and societal challenges.
- UnifiedMC and MAP: Some models, like UnifiedMC and MAP, are marked as temporary, indicating ongoing research and development.
Summary updated