Abstract
Since the release of TÜLU, open resources for instruction tuning have developed quickly, from better base models to new finetuning techniques. We test and incorporate a number of these advances into TÜLU, resulting in TÜLU 2, a suite of improved TÜLU models for advancing the understanding and best practices of adapting pretrained language models to downstream tasks and user preferences. Concretely, we release: (1) TÜLU-V2-mix, an improved collection of high-quality instruction datasets; (2) TÜLU 2, LLAMA-2 models finetuned on the V2 mixture; (3) TÜLU 2+DPO, TÜLU 2 models trained with direct preference optimization (DPO), including the largest DPO-trained model to date (TÜLU 2+DPO 70B); (4) CODE TÜLU 2, CODE LLAMA models finetuned on our V2 mix that outperform CODE LLAMA and its instruction-tuned variant, CODE LLAMA-Instruct. Our evaluation from multiple perspectives shows that the TÜLU 2 suite achieves state-of-the-art performance among open models and matches or exceeds the performance of GPT-3.5-turbo-0301 on several benchmarks.
Introduction
While base models like Llama-2 provide a strong foundation, their practical utility is defined by the "adaptation" process—instruction tuning and alignment with human preferences. However, the optimal recipe for adaptation is a moving target. Since the original TÜLU release, the landscape has shifted with new datasets, specialized base models like Code Llama, and novel alignment algorithms like Direct Preference Optimization (DPO).
TÜLU 2 aims to synthesize these advances into a single, reproducible framework. The project explores critical questions in model adaptation: the optimal mix of human and AI-distilled data, the scalability of DPO to 70B parameter models, and the performance trade-offs of parameter-efficient methods like QLoRA compared to full finetuning.
Methodology
The researchers employed a multi-stage adaptation strategy to create the TÜLU 2 suite:
- Supervised Fine-Tuning (SFT): Llama-2 and Code Llama models were trained on the TÜLU-V2-mix, a curated collection of ~326k samples. This mixture balances human-authored data (FLAN v2, Open Assistant, LIMA) with high-complexity AI-distilled data (ShareGPT, WizardLM, Open-Orca) and domain-specific science literature.
- Direct Preference Optimization (DPO): To align models with human preferences, the team utilized DPO instead of the more complex Reinforcement Learning from Human Feedback (RLHF) via PPO. DPO treats alignment as a classification problem over preferred vs. non-preferred responses, offering greater stability. The UltraFeedback dataset, containing GPT-4 ranked responses, served as the primary preference source.
- Context Expansion: Training context length was increased to 8,192 tokens to improve performance on long-form documents and multi-turn conversations.
- Efficiency Benchmarking: The study compared full-parameter finetuning against QLoRA to evaluate whether quantization and low-rank adaptation can maintain performance levels in high-stakes generation tasks.
Key Results
The TÜLU 2 suite, particularly the 70B parameter variant, represents a significant leap for open-source language models. TÜLU 2+DPO 70B achieves parity with proprietary models like GPT-3.5-turbo-0301 across several benchmarks, including MMLU and Big Bench Hard (BBH).
Beyond general reasoning, CODE TÜLU 2 (34B) demonstrated exceptional performance, significantly outperforming both the base Code Llama and the official Code Llama-Instruct on the Codex-Eval (HumanEval) benchmark.
Insights and Discussion
DPO Scaling and Verbosity
TÜLU 2 provides the first stable demonstration of DPO at the 70B parameter scale. DPO significantly boosts performance in open-ended generation tasks, as seen in AlpacaEval win rates. However, this alignment process introduces a "verbosity bias," where the model tends to produce longer responses to achieve higher win rates—a common trait in RLHF-tuned models.
Data Quality and Efficiency
A key finding was that data quality outweighs quantity. The V2 mixture, despite being smaller than its predecessor, yielded an 8% average performance boost by focusing on high-complexity distilled data and removing lower-quality sets.
Regarding training efficiency, while QLoRA is effective for multiple-choice "academic" benchmarks, it consistently lags behind full finetuning in open-ended, long-form generation. This suggests that for high-quality chat agents, the computational cost of full finetuning remains a worthwhile investment.
Conclusion and Future Work
While TÜLU 2 matches GPT-3.5-turbo in many areas, a substantial gap remains compared to GPT-4. Furthermore, DPO training resulted in a performance dip on multilingual tasks (e.g., TydiQA), likely due to the English-centric nature of the preference data. Future research will focus on improving multilingual alignment and exploring the nuances of safety vs. utility in DPO-trained models.
Summary updated