LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs

LLM-jp, Akiko Aizawa, Eiji Aramaki, Bowen Chen, Fei Cheng, Hiroyuki Deguchi, Rintaro Enomoto, Kazuki Fujii, Kensuke Fukumoto, Takuya Fukushima, Namgi Han, Yuto Harada, Chikara Hashimoto, Tatsuya Hiraoka, Shohei Hisada, Sosuke Hosokawa, Lu Jie, Keisuke Kamata, Teruhito Kanazawa, Hiroki Kanezashi, Hiroshi Kataoka, Satoru Katsumata, Daisuke Kawahara, Seiya Kawano, Atsushi Keyaki, Keisuke Kiryu, Hirokazu Kiyomaru, Takashi Kodama, Takahiro Kubo, Yohei Kuga, Ryoma Kumon, Shuhei Kurita, Sadao Kurohashi, Conglong Li, Taiki Maekawa, Hiroshi Matsuda, Yusuke Miyao, Kentaro Mizuki, Sakae Mizuki, Yugo Murawaki, Akim Mousterou, Ryo Nakamura, Taishi Nakamura, Kouta Nakayama, Tomoka Nakazato, Takuro Niitsuma, Jiro Nishitoba, Yusuke Oda, Hayato Ogawa, Takumi Okamoto, Naoaki Okazaki, Yohei Oseki, Shintaro Ozaki, Koki Ryu, Rafal Rzepka, Keisuke Sakaguchi, Shota Sasaki, Satoshi Sekine, Kohei Suda, Saku Sugawara, Issa Sugiura, Hiroaki Sugiyama, Hisami Suzuki, Jun Suzuki, Toyotaro Suzumura, Kensuke Tachibana, Yu Takagi, Kyosuke Takami, Koichi Takeda, Masashi Takeshita, Masahiro Tanaka, Kenjiro Taura, Arseny Tolmachev, Nobuhiro Ueda, Zhen Wan, Shuntaro Yada, Sakiko Yahata, Yuya Yamamoto, Yusuke Yamauchi, Hitomi Yanaka, Rio Yokota, Koichiro Yoshino

TL;DR

LLM-jp is a massive, open-source collaborative project involving over 1,500 researchers aimed at building high-quality Japanese Large Language Models (LLMs). By releasing everything—from training corpora and tokenizers to model weights and training logs—the project provides a transparent, culturally representative alternative to proprietary "black-box" models.

What Problem Does This Paper Address?

Most state-of-the-art LLMs are developed by a few major organizations with limited transparency regarding their training data and architectures. Furthermore, these models are predominantly English-centric; for instance, Japanese data accounted for only 0.11% of the GPT-3 dataset. This leads to inferior performance in Japanese comprehension and a risk that Japanese culture and intellectual assets could be overshadowed by global English-based standards.

LLM-jp addresses these gaps by creating a "big science" ecosystem in Japan. The project tackles the immense computational and human resource requirements of LLM development through cross-organizational cooperation, ensuring that the resulting models are both high-performing in Japanese and fully open for commercial and non-commercial use.

Methods

The project followed an iterative development cycle, moving from a "learning the ropes" phase to a more sophisticated production phase:

  • Architecture Evolution: The project transitioned from a GPT-2-based architecture for its v1.0 models to the more modern Llama 2 architecture for v2.0 to improve performance and ecosystem compatibility.
  • Tokenization: They developed custom SentencePiece tokenizers. A key improvement in v2.2 was expanding the vocabulary to 100k tokens, which significantly improved tokenization efficiency for Japanese text and code compared to the 50k vocabulary in v2.1.
  • Preprocessing Pipeline: The team built Uzushio, an Apache Spark-based tool for billion-scale corpus cleaning. It handles deduplication via SimHash and applies various rule-based filters (e.g., Hiragana ratio, document length) to ensure data quality.
  • Training Infrastructure: Models were trained on the mdx cloud platform using NVIDIA A100 GPUs. The team had to overcome significant networking challenges, such as packet loss in GPU communication, by tuning RoCE (RDMA over Converged Ethernet) settings.
  • Alignment: The models underwent Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to align outputs with human values.
Figure 1: Timeline of key activities in LLM-jp.

Datasets

LLM-jp utilized a mix of existing and newly created datasets:

  • Pre-training: Used Wikipedia, mC4, the Pile, and the Stack. For v2.0, they extracted a massive 285.5B token Japanese corpus from Common Crawl.
  • Fine-tuning: Utilized jaster (a collection of standard NLP tasks), databricks-dolly-15k, and oasst1/2.
  • Novel Contributions:
    • ichikara-instruction: High-quality Japanese instructions created from scratch by human annotators.
  • AnswerCarefully (AC): A safety-focused dataset of 945 samples designed to help models handle sensitive or harmful prompts appropriately.
  • llm-jp-eval: A comprehensive benchmark tool for evaluating Japanese LLMs across diverse tasks like NLI, QA, and reading comprehension.
  • LLM-jp Toxicity Dataset: 1,867 labeled Japanese texts to facilitate the detection and filtering of toxic content.

Key Insights and Ideas

  • Transparency as a Catalyst: By sharing "discussions and failures" alongside successes, LLM-jp fostered a community that grew from 30 to over 1,500 participants in just one year.
  • The "jaster" Trade-off: While fine-tuning on standard NLP tasks (jaster) boosts scores on traditional benchmarks, it can lead to simplistic, brief responses. Removing these tasks and focusing on conversational data like ichikara significantly improved performance on open-ended benchmarks like MT-Bench.
  • Tokenizer Efficiency Matters: For non-English languages, the vocabulary size and the handling of symbol sequences are critical. Expanding the vocabulary not only improved compression but also prevented the "over-segmentation" of code and English text within the Japanese model.
  • Safety is Culturally Specific: The project emphasizes that what constitutes "harmful" content depends on cultural context, necessitating Japanese-specific safety datasets like AnswerCarefully rather than relying solely on translated English safety sets.

Key Results

The project successfully released two major model suites (v1.0 and v2.0), both featuring 13B parameter models.

  • Performance Gains: The v2.0 models showed substantial improvements over v1.0 across almost all categories in llm-jp-eval.
  • Surpassing GPT-3.5: On the Japanese Vicuna QA benchmark, the v1.1 and v2.0 models achieved higher "AdjustedWinRates" than GPT-3.5 (text-davinci-003).
  • Safety Impact: The inclusion of the AnswerCarefully dataset in v2.0-O reduced the safety violation rate from 68.9% (in the unaligned baseline) to 47.5%.
  • Training Efficiency: The team achieved a processing speed of 170K tokens/second on a 12-node cluster, allowing them to train a 13B model on 270B tokens in roughly 18.4 days.
Table 4: Configurations for the pre-trained models v1.0 and v2.0

Limitations and Future Work

Despite the progress, the authors acknowledge several hurdles:

  • Safety Gap: A 47.5% violation rate is still high for public consumption. More robust red-teaming and larger safety datasets are required.
  • Scaling Up: The current models are 13B parameters; the project is now targeting a 175B-parameter model trained on a 2T token corpus (v3) that includes more Asian languages.
  • Evaluation Depth: Current evaluation frameworks are still incomplete. Future work will involve more detailed analysis of the differences between full parameter tuning and LoRA, as well as the long-term effects of instruction-tuning data mixtures.

Summary updated

Abstract

This paper introduces LLM-jp, a cross-organizational project for the research and development of Japanese large language models (LLMs). LLM-jp aims to develop open-source and strong Japanese LLMs, and as of this writing, more than 1,500 participants from academia and industry are working together for this purpose. This paper presents the background of the establishment of LLM-jp, summaries of its activities, and technical reports on the LLMs developed by LLM-jp. For the latest activities, visit https://llm-jp.nii.ac.jp/en/.
BibTeX
@misc{llmjp2024llm,
  author = {LLM-jp and Akiko Aizawa and Eiji Aramaki and Bowen Chen and Fei Cheng and Hiroyuki Deguchi and Rintaro Enomoto and Kazuki Fujii and Kensuke Fukumoto and Takuya Fukushima and Namgi Han and Yuto Harada and Chikara Hashimoto and Tatsuya Hiraoka and Shohei Hisada and Sosuke Hosokawa and Lu Jie and Keisuke Kamata and Teruhito Kanazawa and Hiroki Kanezashi and Hiroshi Kataoka and Satoru Katsumata and Daisuke Kawahara and Seiya Kawano and Atsushi Keyaki and Keisuke Kiryu and Hirokazu Kiyomaru and Takashi Kodama and Takahiro Kubo and Yohei Kuga and Ryoma Kumon and Shuhei Kurita and Sadao Kurohashi and Conglong Li and Taiki Maekawa and Hiroshi Matsuda and Yusuke Miyao and Kentaro Mizuki and Sakae Mizuki and Yugo Murawaki and Akim Mousterou and Ryo Nakamura and Taishi Nakamura and Kouta Nakayama and Tomoka Nakazato and Takuro Niitsuma and Jiro Nishitoba and Yusuke Oda and Hayato Ogawa and Takumi Okamoto and Naoaki Okazaki and Yohei Oseki and Shintaro Ozaki and Koki Ryu and Rafal Rzepka and Keisuke Sakaguchi and Shota Sasaki and Satoshi Sekine and Kohei Suda and Saku Sugawara and Issa Sugiura and Hiroaki Sugiyama and Hisami Suzuki and Jun Suzuki and Toyotaro Suzumura and Kensuke Tachibana and Yu Takagi and Kyosuke Takami and Koichi Takeda and Masashi Takeshita and Masahiro Tanaka and Kenjiro Taura and Arseny Tolmachev and Nobuhiro Ueda and Zhen Wan and Shuntaro Yada and Sakiko Yahata and Yuya Yamamoto and Yusuke Yamauchi and Hitomi Yanaka and Rio Yokota and Koichiro Yoshino},
  title = {{LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs}},
  year = {2024},
  eprint = {2407.03963},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.03963},
}
Figure