TL;DR
LLM-jp is a massive, open-source collaborative project involving over 1,500 researchers aimed at building high-quality Japanese Large Language Models (LLMs). By releasing everything—from training corpora and tokenizers to model weights and training logs—the project provides a transparent, culturally representative alternative to proprietary "black-box" models.
What Problem Does This Paper Address?
Most state-of-the-art LLMs are developed by a few major organizations with limited transparency regarding their training data and architectures. Furthermore, these models are predominantly English-centric; for instance, Japanese data accounted for only 0.11% of the GPT-3 dataset. This leads to inferior performance in Japanese comprehension and a risk that Japanese culture and intellectual assets could be overshadowed by global English-based standards.
LLM-jp addresses these gaps by creating a "big science" ecosystem in Japan. The project tackles the immense computational and human resource requirements of LLM development through cross-organizational cooperation, ensuring that the resulting models are both high-performing in Japanese and fully open for commercial and non-commercial use.
Methods
The project followed an iterative development cycle, moving from a "learning the ropes" phase to a more sophisticated production phase:
- Architecture Evolution: The project transitioned from a GPT-2-based architecture for its v1.0 models to the more modern Llama 2 architecture for v2.0 to improve performance and ecosystem compatibility.
- Tokenization: They developed custom SentencePiece tokenizers. A key improvement in v2.2 was expanding the vocabulary to 100k tokens, which significantly improved tokenization efficiency for Japanese text and code compared to the 50k vocabulary in v2.1.
- Preprocessing Pipeline: The team built Uzushio, an Apache Spark-based tool for billion-scale corpus cleaning. It handles deduplication via SimHash and applies various rule-based filters (e.g., Hiragana ratio, document length) to ensure data quality.
- Training Infrastructure: Models were trained on the mdx cloud platform using NVIDIA A100 GPUs. The team had to overcome significant networking challenges, such as packet loss in GPU communication, by tuning RoCE (RDMA over Converged Ethernet) settings.
- Alignment: The models underwent Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to align outputs with human values.
Datasets
LLM-jp utilized a mix of existing and newly created datasets:
- Pre-training: Used Wikipedia, mC4, the Pile, and the Stack. For v2.0, they extracted a massive 285.5B token Japanese corpus from Common Crawl.
- Fine-tuning: Utilized
jaster(a collection of standard NLP tasks),databricks-dolly-15k, andoasst1/2. - Novel Contributions:
ichikara-instruction: High-quality Japanese instructions created from scratch by human annotators.
AnswerCarefully(AC): A safety-focused dataset of 945 samples designed to help models handle sensitive or harmful prompts appropriately.llm-jp-eval: A comprehensive benchmark tool for evaluating Japanese LLMs across diverse tasks like NLI, QA, and reading comprehension.- LLM-jp Toxicity Dataset: 1,867 labeled Japanese texts to facilitate the detection and filtering of toxic content.
Key Insights and Ideas
- Transparency as a Catalyst: By sharing "discussions and failures" alongside successes, LLM-jp fostered a community that grew from 30 to over 1,500 participants in just one year.
- The "jaster" Trade-off: While fine-tuning on standard NLP tasks (
jaster) boosts scores on traditional benchmarks, it can lead to simplistic, brief responses. Removing these tasks and focusing on conversational data likeichikarasignificantly improved performance on open-ended benchmarks like MT-Bench. - Tokenizer Efficiency Matters: For non-English languages, the vocabulary size and the handling of symbol sequences are critical. Expanding the vocabulary not only improved compression but also prevented the "over-segmentation" of code and English text within the Japanese model.
- Safety is Culturally Specific: The project emphasizes that what constitutes "harmful" content depends on cultural context, necessitating Japanese-specific safety datasets like
AnswerCarefullyrather than relying solely on translated English safety sets.
Key Results
The project successfully released two major model suites (v1.0 and v2.0), both featuring 13B parameter models.
- Performance Gains: The v2.0 models showed substantial improvements over v1.0 across almost all categories in
llm-jp-eval. - Surpassing GPT-3.5: On the Japanese Vicuna QA benchmark, the v1.1 and v2.0 models achieved higher "AdjustedWinRates" than GPT-3.5 (
text-davinci-003). - Safety Impact: The inclusion of the
AnswerCarefullydataset in v2.0-O reduced the safety violation rate from 68.9% (in the unaligned baseline) to 47.5%. - Training Efficiency: The team achieved a processing speed of 170K tokens/second on a 12-node cluster, allowing them to train a 13B model on 270B tokens in roughly 18.4 days.
Limitations and Future Work
Despite the progress, the authors acknowledge several hurdles:
- Safety Gap: A 47.5% violation rate is still high for public consumption. More robust red-teaming and larger safety datasets are required.
- Scaling Up: The current models are 13B parameters; the project is now targeting a 175B-parameter model trained on a 2T token corpus (v3) that includes more Asian languages.
- Evaluation Depth: Current evaluation frameworks are still incomplete. Future work will involve more detailed analysis of the differences between full parameter tuning and LoRA, as well as the long-term effects of instruction-tuning data mixtures.
Summary updated