TL;DR
Fine-tuning large language models (LLMs) is memory-intensive. This paper introduces HELENE, a novel zeroth-order optimization method that significantly speeds up LLM fine-tuning by intelligently using Hessian information and layer-wise clipping. HELENE achieves up to a 20x speedup and improves accuracy, making LLM fine-tuning more efficient and accessible.
What Problem Does This Paper Address?
Fine-tuning massive LLMs is a critical step for adapting them to specific tasks, but it's notoriously memory-hungry due to the backpropagation process. While zeroth-order (ZO) optimization methods like MeZO reduce memory by avoiding backpropagation, they suffer from slow convergence. This slowness is attributed to the varying "curvatures" (how quickly the loss function changes) across different parts of the LLM's parameters. Standard ZO methods struggle to adapt to these complex landscapes, leading to inefficient training. This paper aims to bridge this gap by developing a memory-efficient ZO optimizer that leverages second-order information (related to curvature) to accelerate convergence without the prohibitive memory costs of traditional second-order methods.
Methods
The core of HELENE lies in its novel approach to incorporating second-order information within a zeroth-order framework. Key techniques include:
- Asymptotic Gauss-Newton-Bartlett (A-GNB) Estimator: This is a new method for estimating the diagonal of the Hessian matrix. Unlike previous methods that might rely on sampled labels (which can be noisy, especially in imbalanced datasets like LLMs), A-GNB uses true labels to provide a more accurate and less noisy Hessian approximation. This estimator is designed to asymptotically converge to the true Hessian.
- Layer-wise Adaptive Clipping: Instead of applying a single clipping threshold to the entire Hessian (which can be suboptimal), HELENE applies clipping on a per-layer basis. This allows the optimizer to adapt its updates based on the specific curvature of each layer, preventing extreme values in some layers from disproportionately affecting the entire model's update. This is a key innovation for handling heterogeneous curvatures.
- Gradient Annealing: HELENE incorporates an annealing mechanism for the exponential moving average (EMA) of gradients. This helps to dynamically adjust the learning rate, reducing the influence of noisy or outdated gradients in later stages of training and improving stability in non-convex landscapes.
These components are combined to create an optimizer that is both memory-efficient (like MeZO) and faster-converging (by incorporating curvature information).
Datasets
The paper evaluates HELENE on:
- RoBERTa-large: A BERT-based masked language model.
- OPT-1.3B: A decoder-only large language model.
Experiments are conducted across various NLP tasks, including sentiment classification, natural language inference, topic classification, multiple choice, and text generation. The paper also mentions using datasets from the SuperGLUE benchmark.
Key Insights and Ideas
- Curvature Matters, Even in ZO: The paper strongly emphasizes that the varying curvature of LLM loss landscapes is a major bottleneck for zeroth-order optimization. Simply reducing memory isn't enough; adapting to the model's internal geometry is crucial for speed.
- Layer-wise Adaptability is Key: Applying optimization strategies (like clipping) globally can be detrimental. HELENE's layer-wise approach allows it to tailor updates to the specific characteristics of each part of the LLM, leading to more robust and efficient training.
- True Labels for Better Hessian: Using true labels instead of sampled ones for Hessian estimation (as in A-GNB) significantly reduces noise and improves the accuracy of curvature information, which is vital for effective second-order pre-conditioning.
- Theoretical Guarantees for Scalability: The theoretical analysis showing that HELENE's convergence scales with the largest layer dimension rather than the total parameter dimension is a significant insight. This makes it particularly well-suited for modern, deep LLM architectures where layer dimensions can vary greatly.
Key Results
- Speedup: HELENE achieves up to a 20x speedup compared to MeZO.
- Accuracy Improvement: On average, HELENE shows an accuracy improvement of 1.5% across tasks.
- Compatibility: HELENE works seamlessly with both full-parameter fine-tuning (FT) and parameter-efficient fine-tuning (PEFT) methods like LoRA and prefix-tuning.
- Performance: HELENE consistently outperforms MeZO and other state-of-the-art optimizers on various tasks and datasets.
- Memory Efficiency: While using second-order information, HELENE remains memory-efficient, requiring only 14GB for OPT-1.3B fine-tuning, which is comparable to or better than other methods.
Limitations and Future Work
- Memory Usage: Like other second-order optimizers, HELENE stores history of gradients and Hessian values, leading to memory usage proportional to the number of parameters. While more efficient than some, it still requires more memory than first-order methods.
- Hyperparameter Tuning: The paper mentions hyperparameters like λ (for clipping) and γ, β1, β2 (for EMA), which would require tuning.
- Future Work: The paper's theoretical analysis suggests that HELENE is highly scalable for modern LLM architectures. Future work could involve exploring its application to even larger models and a wider range of tasks, potentially investigating more advanced Hessian approximation techniques or adaptive clipping strategies.
Summary updated