Memory-Efficient Backpropagation for Fine-Tuning LLMs on Resource-Constrained Mobile Devices

Congzheng Song, Xinyu TangEMNLP 2025

TL;DR

This paper introduces Memory-Efficient Backpropagation (MeBP), a novel technique that enables fine-tuning of large language models (LLMs) on resource-constrained mobile devices. MeBP achieves a better trade-off between memory usage and computation time compared to existing methods, allowing LLMs up to 4B parameters to be fine-tuned using less than 1GB of memory.

What Problem Does This Paper Address?

Fine-tuning large language models (LLMs) is crucial for personalizing them to specific user data and tasks. However, the memory demands of backpropagation, even with parameter-efficient methods like LoRA, are prohibitively high for mobile devices. While alternative methods like zeroth-order optimization (ZO) reduce memory footprints, they suffer from significantly slower convergence and poorer performance. This creates a gap where on-device fine-tuning for enhanced LLM utility is desired but practically infeasible due to memory and computational constraints on mobile hardware.

Methods

The core of MeBP lies in a memory-efficient implementation of backpropagation, built upon gradient checkpointing. Key optimizations include:

  • Lazy Weight Loading and Decompression: Base model weights are memory-mapped and only decompressed on demand when needed for computation, rather than being loaded and decompressed upfront. This drastically reduces peak memory usage.
  • Memory-Mapped Activation Checkpoints: Intermediate activations are also memory-mapped, preventing them from occupying precious RAM.
  • Gradient Checkpointing Compilation: The LLM is split into blocks (e.g., transformer layers), and backpropagation is performed on a block-by-block basis. This trades increased computation for reduced memory by recomputing activations during the backward pass.
  • LoRA Fine-tuning: The method focuses on fine-tuning LoRA adapters, which significantly reduces the number of trainable parameters and thus the memory required for gradients and optimizer states.
  • Base Model Weights Compression: 4-bit INT4 quantization is applied to non-LoRA parameters to reduce disk space.

Datasets

  • WikiText-2: Used for language modeling tasks.
  • ToolACE: Used for function calling tasks.
  • Berkeley Function Calling (BFCL) v4: Also used for function calling tasks, specifically a live split for more realistic scenarios.

Key Insights and Ideas

  • Backpropagation is Feasible on Mobile: The paper demonstrates that with careful memory management, traditional backpropagation (via MeBP) can be made practical for mobile devices, offering a superior trade-off to ZO methods.
  • Lazy Loading is Key: The strategy of lazily decompressing and loading weights and activations is a critical enabler for fitting LLM fine-tuning into tight memory budgets.
  • LoRA is a Good Fit: Combining MeBP with LoRA is a powerful strategy, as LoRA already reduces the parameter count, making the memory savings from MeBP even more impactful.
  • ZO's Trade-offs are Significant: The experiments clearly show that while ZO methods save memory, their slow convergence and performance degradation make them less practical for achieving good results in a reasonable time on mobile.

Key Results

  • Memory Efficiency: MeBP allows fine-tuning of LLMs up to 4B parameters using less than 1GB of memory on an iPhone 15 Pro Max. This is a significant reduction compared to previous mobile LLM fine-tuning efforts.
  • Faster Convergence: MeBP converges significantly faster than MeZO (a leading ZO method) in terms of both the number of optimization steps and total wall-clock time. For example, in language modeling, MeBP achieves substantial improvements within the first 100 steps, while MeZO requires over 1,000 steps for similar gains.
  • Comparable Performance: MeBP achieves comparable performance to full parameter fine-tuning (FFT) on certain tasks using LoRA, validating the approach.
  • Decompression Overhead: While decompression adds overhead (32-42% of forward pass time, 13-24% of backward pass time), the memory savings are substantial, making it a worthwhile trade-off.
  • Sequence Length Impact: Shorter sequence lengths are more suitable for current MeBP implementations on mobile due to memory and computation constraints.

Limitations and Future Work

  • Device Availability: MeBP has only been verified on iOS using an iPhone 15 Pro Max. Performance may vary on other mobile operating systems or hardware configurations.
  • Sequence Length: The current implementation does not scale well with sequence length, limiting its applicability to data types with shorter sequences. Future work could explore efficient fine-tuning on longer sequences.
  • Hardware Specificity: For language modeling tasks, MeBP encounters a bottleneck at the final layer due to large matrix multiplications. Future work could explore hardware-specific optimizations or techniques like fused kernels or sampled softmax.
  • More Aggressive Compression: The paper mentions investigating more aggressive compression methods (e.g., 2-bit quantization-aware training) as future work.

Summary updated

BibTeX
@inproceedings{song2025memory,
  author = {Congzheng Song and Xinyu Tang},
  title = {{Memory-Efficient Backpropagation for Fine-Tuning LLMs on Resource-Constrained Mobile Devices}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure