SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

Sanghoon Kim, Dahyun Kim, Chanjun Park, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, Sunghun KimNAACL 2024 · DOI

TL;DR

This paper introduces SOLAR 10.7B, a 10.7 billion parameter LLM, and a novel "depth up-scaling" (DUS) method that efficiently scales LLMs by increasing their depth (number of layers) and continuing pre-training. SOLAR 10.7B demonstrates superior performance across various NLP tasks, and its instruction-tuned variant, SOLAR 10.7B-Instruct, outperforms even larger models like Mixtral-8x7B-Instruct.

What Problem Does This Paper Address?

The rapid advancement of Large Language Models (LLMs) has been driven by a "scaling law" that shows performance improves with model size and training data. This has led to a trend of building ever-larger models, which presents significant challenges in terms of computational cost for training and inference, as well as accessibility. While methods like Mixture-of-Experts (MoE) offer a way to scale models efficiently, they often require complex modifications to training and inference frameworks, hindering widespread adoption. The authors aim to address this by proposing a simpler, yet effective, method for scaling LLMs that maintains computational efficiency and compatibility with existing tools.

Methods

The core innovation is the Depth Up-Scaling (DUS) method, a two-stage process:

  1. Depthwise Scaling: This involves taking a base LLM (e.g., a 32-layer model) and creating a larger model by duplicating layers. Specifically, they remove the last m layers from the original model and the first m layers from a duplicate, then concatenate them to form a model with s = 2 * (n - m) layers. For instance, they scale a 32-layer model to 48 layers by removing 8 layers from each. This is inspired by Tan and Le (2019) but adapted for LLMs.
  2. Continued Pretraining: After depthwise scaling, the model's performance initially drops. To recover and improve performance, they continue pretraining the scaled model. The authors hypothesize that the specific way DUS scales depth isolates heterogeneity in the model, allowing for faster performance recovery during this stage.

They also introduce SOLAR 10.7B-Instruct, a variant fine-tuned for instruction following, using a two-stage process: instruction tuning and alignment tuning (using Direct Preference Optimization - DPO).

Datasets

The paper utilizes a combination of publicly available and synthesized datasets:

  • Pretraining: The base SOLAR 10.7B model is pretrained.
  • Instruction Tuning: Alpaca-GPT4, OpenOrca, and a newly synthesized "Synth. Math-Instruct" dataset (created by rephrasing math questions and answers from the Math dataset) are used.
  • Alignment Tuning: Orca DPO Pairs, Ultrafeedback Cleaned, and a newly synthesized "Synth. Math-Alignment" dataset (derived from "Synth. Math-Instruct") are used.

Key Insights and Ideas

  • Simplicity is Key: DUS offers a straightforward way to scale LLMs without the need for complex architectural changes like MoE, making it highly compatible with existing LLM frameworks.
  • Depthwise Scaling Works: The paper demonstrates that increasing the depth of a model, combined with continued pretraining, is an effective strategy for scaling performance.
  • Continued Pretraining is Crucial: Simply scaling the model's depth isn't enough; continued pretraining is essential for performance recovery and improvement.
  • Heterogeneity Aids Recovery: The authors hypothesize that DUS's method of depthwise scaling creates a specific type of heterogeneity that facilitates rapid performance recovery during continued pretraining.
  • Instruction Tuning Matters: The SOLAR 10.7B-Instruct variant shows significant improvements in following instructions, highlighting the importance of fine-tuning for specific capabilities.

Key Results

  • SOLAR 10.7B: Outperforms other pretrained models of similar size (e.g., Qwen 14B, Mistral 7B) on various benchmarks.
  • SOLAR 10.7B-Instruct: Achieves the highest H6 score (average across six tasks) among evaluated models, even surpassing the larger Mixtral-8x7B-Instruct-v0.1 and Qwen 72B.
  • DUS Effectiveness: The DUS method is shown to be effective in scaling models that can achieve state-of-the-art performance when fine-tuned.
  • Data Contamination: SOLAR 10.7B-Instruct shows low levels of data contamination across tested benchmarks, indicating robust training practices.

Limitations and Future Work

  • Hyperparameter Optimization: The paper acknowledges that the choice of m=8 layers removed during depthwise scaling was primarily due to hardware limitations and not exhaustively optimized. Further exploration of hyperparameters in DUS is needed.
  • Computational Cost: While DUS is computationally efficient for scaling, training and inference of LLMs still demand significant resources, which might limit accessibility for some.
  • Environmental Impact: The substantial energy consumption for training and operating LLMs raises environmental concerns.
  • Task-Specific Fine-tuning: While SOLAR 10.7B-Instruct is good at following instructions, it may still require task-specific fine-tuning for optimal performance in specialized applications.
  • Future Work: The authors suggest further exploration of DUS, including comparative analyses of different depthwise scaling strategies and more comprehensive hyperparameter tuning. They also aim to address the computational demands and environmental impact of LLMs.

Summary updated

BibTeX
@inproceedings{kim2024solar,
  author = {Sanghoon Kim and Dahyun Kim and Chanjun Park and Wonsung Lee and Wonho Song and Yunsu Kim and Hyeonwoo Kim and Yungi Kim and Hyeonju Lee and Jihoo Kim and Changbae Ahn and Seonghoon Yang and Sukyung Lee and Hyunbyung Park and Gyoungjin Gim and Mikyoung Cha and Hwalsuk Lee and Sunghun Kim},
  title = {{SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-industry.3},
}
Figure