Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2

Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew E. Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, Hannaneh HajishirziarXiv 2023 · OpenReview

Abstract

Since the release of TÜLU, open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into TÜLU, resulting in TÜLU 2, a suite of improved TÜLU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) TÜLU-V2-mix, an improved collection of high-quality instruction datasets; (2) TÜLU 2, LLAMA-2 models finetuned on the V2 mixture; (3) TÜLU 2+DPO, TÜLU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (TÜLU 2+DPO 70B); (4) CODE TÜLU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the TÜLU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks.

Introduction

While base models like Llama-2 provide a strong foundation, their practical utility is defined by the "adaptation" process—instruction tuning and alignment with human preferences. However, the optimal recipe for adaptation is a moving target. Since the original TÜLU release, the landscape has shifted with new datasets, specialized base models like Code Llama, and novel alignment algorithms like Direct Preference Optimization (DPO).

TÜLU 2 aims to synthesize these advances into a single, reproducible framework. The project explores critical questions in model adaptation: the optimal mix of human and AI-distilled data, the scalability of DPO to 70B parameter models, and the performance trade-offs of parameter-efficient methods like QLoRA compared to full finetuning.

Methodology

The researchers employed a multi-stage adaptation strategy to create the TÜLU 2 suite:

  1. Supervised Fine-Tuning (SFT): Llama-2 and Code Llama models were trained on the TÜLU-V2-mix, a curated collection of ~326k samples. This mixture balances human-authored data (FLAN v2, Open Assistant, LIMA) with high-complexity AI-distilled data (ShareGPT, WizardLM, Open-Orca) and domain-specific science literature.
  2. Direct Preference Optimization (DPO): To align models with human preferences, the team utilized DPO instead of the more complex Reinforcement Learning from Human Feedback (RLHF) via PPO. DPO treats alignment as a classification problem over preferred vs. non-preferred responses, offering greater stability. The UltraFeedback dataset, containing GPT-4 ranked responses, served as the primary preference source.
  3. Context Expansion: Training context length was increased to 8,192 tokens to improve performance on long-form documents and multi-turn conversations.
  4. Efficiency Benchmarking: The study compared full-parameter finetuning against QLoRA to evaluate whether quantization and low-rank adaptation can maintain performance levels in high-stakes generation tasks.

Key Results

The TÜLU 2 suite, particularly the 70B parameter variant, represents a significant leap for open-source language models. TÜLU 2+DPO 70B achieves parity with proprietary models like GPT-3.5-turbo-0301 across several benchmarks, including MMLU and Big Bench Hard (BBH).

Performance of TÜLU 2 suite compared to proprietary and other open models across standard benchmarks.

Beyond general reasoning, CODE TÜLU 2 (34B) demonstrated exceptional performance, significantly outperforming both the base Code Llama and the official Code Llama-Instruct on the Codex-Eval (HumanEval) benchmark.

Insights and Discussion

DPO Scaling and Verbosity

TÜLU 2 provides the first stable demonstration of DPO at the 70B parameter scale. DPO significantly boosts performance in open-ended generation tasks, as seen in AlpacaEval win rates. However, this alignment process introduces a "verbosity bias," where the model tends to produce longer responses to achieve higher win rates—a common trait in RLHF-tuned models.

Comparison of generation performance (MT-Bench, AlpacaEval) and average output length across model sizes.

Data Quality and Efficiency

A key finding was that data quality outweighs quantity. The V2 mixture, despite being smaller than its predecessor, yielded an 8% average performance boost by focusing on high-complexity distilled data and removing lower-quality sets.

Regarding training efficiency, while QLoRA is effective for multiple-choice "academic" benchmarks, it consistently lags behind full finetuning in open-ended, long-form generation. This suggests that for high-quality chat agents, the computational cost of full finetuning remains a worthwhile investment.

Conclusion and Future Work

While TÜLU 2 matches GPT-3.5-turbo in many areas, a substantial gap remains compared to GPT-4. Furthermore, DPO training resulted in a performance dip on multilingual tasks (e.g., TydiQA), likely due to the English-centric nature of the preference data. Future research will focus on improving multilingual alignment and exploring the nuances of safety vs. utility in DPO-trained models.

Summary updated

Abstract

Since the release of T\"ULU [Wang et al., 2023b], open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into T\"ULU, resulting in T\"ULU 2, a suite of improved T\"ULU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) T\"ULU-V2-mix, an improved collection of high-quality instruction datasets; (2) T\"ULU 2, LLAMA-2 models finetuned on the V2 mixture; (3) T\"ULU 2+DPO, T\"ULU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (T\"ULU 2+DPO 70B); (4) CODE T\"ULU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the T\"ULU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks. We release all the checkpoints, data, training and evaluation code to facilitate future open efforts on adapting large language models.
BibTeX
@misc{ivison2023camels,
  author = {Hamish Ivison and Yizhong Wang and Valentina Pyatkin and Nathan Lambert and Matthew E. Peters and Pradeep Dasigi and Joel Jang and David Wadden and Noah A. Smith and Iz Beltagy and Hannaneh Hajishirzi},
  title = {{Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2}},
  year = {2023},
  eprint = {2311.10702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.10702},
}
Figure