TL;DR
This paper introduces Memory-Efficient Backpropagation (MeBP), a novel technique that enables fine-tuning of large language models (LLMs) on resource-constrained mobile devices. MeBP achieves a better trade-off between memory usage and computation time compared to existing methods, allowing LLMs up to 4B parameters to be fine-tuned using less than 1GB of memory.
What Problem Does This Paper Address?
Fine-tuning large language models (LLMs) is crucial for personalizing them to specific user data and tasks. However, the memory demands of backpropagation, even with parameter-efficient methods like LoRA, are prohibitively high for mobile devices. While alternative methods like zeroth-order optimization (ZO) reduce memory footprints, they suffer from significantly slower convergence and poorer performance. This creates a gap where on-device fine-tuning for enhanced LLM utility is desired but practically infeasible due to memory and computational constraints on mobile hardware.
Methods
The core of MeBP lies in a memory-efficient implementation of backpropagation, built upon gradient checkpointing. Key optimizations include:
- Lazy Weight Loading and Decompression: Base model weights are memory-mapped and only decompressed on demand when needed for computation, rather than being loaded and decompressed upfront. This drastically reduces peak memory usage.
- Memory-Mapped Activation Checkpoints: Intermediate activations are also memory-mapped, preventing them from occupying precious RAM.
- Gradient Checkpointing Compilation: The LLM is split into blocks (e.g., transformer layers), and backpropagation is performed on a block-by-block basis. This trades increased computation for reduced memory by recomputing activations during the backward pass.
- LoRA Fine-tuning: The method focuses on fine-tuning LoRA adapters, which significantly reduces the number of trainable parameters and thus the memory required for gradients and optimizer states.
- Base Model Weights Compression: 4-bit INT4 quantization is applied to non-LoRA parameters to reduce disk space.
Datasets
- WikiText-2: Used for language modeling tasks.
- ToolACE: Used for function calling tasks.
- Berkeley Function Calling (BFCL) v4: Also used for function calling tasks, specifically a live split for more realistic scenarios.
Key Insights and Ideas
- Backpropagation is Feasible on Mobile: The paper demonstrates that with careful memory management, traditional backpropagation (via MeBP) can be made practical for mobile devices, offering a superior trade-off to ZO methods.
- Lazy Loading is Key: The strategy of lazily decompressing and loading weights and activations is a critical enabler for fitting LLM fine-tuning into tight memory budgets.
- LoRA is a Good Fit: Combining MeBP with LoRA is a powerful strategy, as LoRA already reduces the parameter count, making the memory savings from MeBP even more impactful.
- ZO's Trade-offs are Significant: The experiments clearly show that while ZO methods save memory, their slow convergence and performance degradation make them less practical for achieving good results in a reasonable time on mobile.
Key Results
- Memory Efficiency: MeBP allows fine-tuning of LLMs up to 4B parameters using less than 1GB of memory on an iPhone 15 Pro Max. This is a significant reduction compared to previous mobile LLM fine-tuning efforts.
- Faster Convergence: MeBP converges significantly faster than MeZO (a leading ZO method) in terms of both the number of optimization steps and total wall-clock time. For example, in language modeling, MeBP achieves substantial improvements within the first 100 steps, while MeZO requires over 1,000 steps for similar gains.
- Comparable Performance: MeBP achieves comparable performance to full parameter fine-tuning (FFT) on certain tasks using LoRA, validating the approach.
- Decompression Overhead: While decompression adds overhead (32-42% of forward pass time, 13-24% of backward pass time), the memory savings are substantial, making it a worthwhile trade-off.
- Sequence Length Impact: Shorter sequence lengths are more suitable for current MeBP implementations on mobile due to memory and computation constraints.
Limitations and Future Work
- Device Availability: MeBP has only been verified on iOS using an iPhone 15 Pro Max. Performance may vary on other mobile operating systems or hardware configurations.
- Sequence Length: The current implementation does not scale well with sequence length, limiting its applicability to data types with shorter sequences. Future work could explore efficient fine-tuning on longer sequences.
- Hardware Specificity: For language modeling tasks, MeBP encounters a bottleneck at the final layer due to large matrix multiplications. Future work could explore hardware-specific optimizations or techniques like fused kernels or sampled softmax.
- More Aggressive Compression: The paper mentions investigating more aggressive compression methods (e.g., 2-bit quantization-aware training) as future work.
Summary updated