TL;DR
This paper introduces TeleChat, a new family of open-source bilingual (English/Chinese) large language models (LLMs) with 7B and 12B parameters. TeleChat achieves state-of-the-art performance on various benchmarks, particularly excelling in general dialogue, knowledge-based Q&A, and demonstrating a significantly extended context window of 96k tokens.
What Problem Does This Paper Address?
The LLM landscape is rapidly evolving, with many powerful models remaining proprietary. While open-source alternatives are emerging, there's a notable lack of transparency regarding their instruction fine-tuning processes, including details about training data, multi-turn dialogue handling, and techniques for improving conversational abilities. Furthermore, existing open-source LLMs often have limited context windows, hindering their ability to process and understand long documents or conversations. TeleChat aims to address these gaps by providing a transparent, high-performing, bilingual LLM with an extended context window, along with its checkpoints and code for the community.
Methods
TeleChat builds upon the autoregressive transformer architecture, drawing inspiration from models like GPT-3, LLaMA, and BLOOM. Key architectural choices include:
- Rotary Positional Embedding (RoPE): Used for encoding absolute positions with explicit relative position dependencies.
- Flash Attention v2: Implemented for improved computational efficiency and reduced memory usage in attention modules.
- RMSNorm: Replaces conventional layer normalization for enhanced training stability and performance.
- SwiGLU Activations: Employed in the feed-forward networks.
A significant contribution is the extended context window of up to 96k tokens. This is achieved through:
- NTK-aware interpolation: Redistributes interpolation pressure across dimensions.
- Dynamic NTK-aware interpolation: Dynamically adjusts the interpolation scaling factor based on real-time context length.
- Multi-stage Long-context Training: Periodically extends context length during supervised fine-tuning.
- Attention-Scaling: Rescales dot products relative to the context-to-training length ratio during inference.
The fine-tuning process involves:
- Supervised Fine-Tuning (SFT): Using a carefully curated instruction dataset.
- Noisy Embedding Fine-Tuning (NEFTUNE): Adding random noise to input embeddings to potentially improve performance, especially with smaller datasets.
- Reinforcement Learning (RL): Using Proximal Policy Optimization (PPO) to align model outputs with human preferences for safety and norms.
- Knowledge Graph Integration: To mitigate hallucinations by providing structured, factual information.
Datasets
- Pretraining Corpus: A massive, diverse corpus of trillions of tokens, including English and Chinese texts from web pages, social platforms, encyclopedias, books, academic papers, and code repositories. The paper highlights this as the largest open Chinese corpus for language model pre-training to date.
- Supervised Fine-Tuning Data: Over 100,000 samples collected through manual annotation by native Chinese speakers across various academic backgrounds.
- Reward Model Data: Collected from human annotation and internal user testing, with a focus on high-quality and diverse prompts.
- Evaluation Benchmarks: A comprehensive suite of standard benchmarks including MMLU, C-Eval, CMMLU, AGIEval, GAOKAO-Bench, CSL, EPRSTMT, CHID, GSM8K, Math, and HumanEval. The paper also includes evaluations on general dialogue Q&A, safety, and hallucination tasks.
Key Insights and Ideas
- Bilingualism is Key: The focus on both English and Chinese is crucial for broader applicability, especially in a global context.
- Extended Context is Achievable: The paper demonstrates that achieving very long context windows (96k) is feasible for open-source models without prohibitive performance degradation, using techniques like NTK-aware interpolation and multi-stage training.
- Transparency Matters: The authors emphasize the importance of disclosing fine-tuning methodologies and data, a common gap in other prominent LLMs.
- NEFTUNE's Diminishing Returns: While NEFTUNE shows promise with smaller datasets (e.g., 10,000 samples), its advantage significantly decreases with larger fine-tuning datasets, suggesting models become less prone to overfitting.
- Knowledge Graphs Enhance Factuality: Integrating knowledge graphs demonstrably improves accuracy on factual Q&A tasks, reducing hallucinations.
Key Results
- State-of-the-Art Performance: TeleChat-7B and TeleChat-12B achieve state-of-the-art results among open-source models of similar sizes across a wide range of public benchmarks.
- Extended Context Window: Successfully extends context window to over 96k tokens, the longest among open-source LLMs.
- General Dialogue Performance: TeleChat-12B achieves a 99.3% performance level compared to GPT-3.5 on general dialogue tasks.
- Factual Q&A Improvement: Accuracy on a KG-based Q&A task improves from 0.19 to 0.69 with knowledge graph integration.
Limitations and Future Work
- Computational Resources: The extensive computational resources required for training and inference may pose challenges for wider adoption.
- Knowledge Graph Completeness: The effectiveness of KG integration is dependent on the completeness and up-to-dateness of the underlying knowledge graph data, which can introduce biases or inaccuracies.
- Real-World Applicability: While evaluated on many benchmarks, the paper acknowledges that the evaluation may not fully capture all real-world applicability and limitations in specific domains or scenarios.
Future work will likely focus on addressing these limitations, potentially exploring more efficient training methods, improving KG integration, and conducting more extensive real-world application testing.
Summary updated