TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

Han Cai, Chuang Gan, Ligeng Zhu, Song Han

Abstract

On-device learning enables edge devices to continually adapt AI models to new data, which requires a small memory footprint to fit the tight memory constraints of edge hardware. Existing work often attempts to solve this problem by reducing the number of trainable parameters. However, this does not directly translate to memory savings since the major bottleneck is the activations, not the parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes weights while only learning bias modules, thus eliminating the need to store intermediate activations. To maintain adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps, adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.5-12.9x memory saving without sacrificing accuracy compared to fine-tuning a full Inception-V3.

The Activation Bottleneck

On-device learning is notoriously difficult because training a neural network requires far more memory than inference. The primary culprit is not the number of parameters, but the activations. During standard backpropagation, the network must store intermediate feature maps (activations) to calculate gradients for the weights.

In modern efficient architectures like MobileNetV2, activations can be nearly 14x larger than the parameters themselves during training. Consequently, training can require over 1GB of RAM—far exceeding the capacity of many low-end microcontrollers or the tiny on-chip SRAM of mobile CPUs. Reducing parameters alone is a "false economy" for on-device memory if the activation footprint remains high.

Methodology: Tiny-Transfer-Learning

TinyTL shifts the focus from parameter-efficiency to memory-efficiency through three main innovations:

  1. Weight Freezing & Bias Tuning: The authors observed that the gradient of the loss with respect to weights requires storing activations, but the gradient with respect to biases does not. By freezing the weights of the feature extractor and only updating the biases, TinyTL eliminates the need to store most intermediate activations.
  2. Lite Residual Modules: Simply tuning biases is often insufficient for complex new tasks. To restore learning capacity, TinyTL adds small, memory-efficient residual blocks. These modules use group convolutions and low-resolution inputs (via 2x2 downsampling) to keep their own activation overhead minimal (approximately 3.8% of total memory).
  3. Feature Extractor Adaptation: Since the main weights are frozen, selecting the right starting model is crucial. TinyTL uses a Once-for-All (OFA) network to automatically select a specialized sub-network backbone that best fits the specific target dataset before training begins.
Comparison of fine-tuning strategies: (a) full network, (b) bias only, and (c) TinyTL's lite residual learning.

Experimental Results

The researchers evaluated TinyTL across a wide variety of computer vision tasks, including Oxford Flowers, Stanford Cars, and CIFAR-100.

  • Massive Memory Savings: TinyTL achieves 7.3x to 12.9x memory reduction compared to fine-tuning a full Inception-V3 model while matching its accuracy.
  • Superior Accuracy: Compared to the common practice of only fine-tuning the "last layer" of a network, TinyTL improves accuracy by up to 34.1% on challenging datasets like Stanford Cars.
  • SRAM-Only Training: By shrinking the training footprint to around 16MB (at batch size 1), TinyTL makes it possible to train models entirely within a CPU's L3 cache. This avoids slow and power-hungry DRAM access, potentially revolutionizing the energy efficiency of on-device AI.
Accuracy vs. training memory footprint across eight downstream transfer learning tasks.

Conclusion

TinyTL demonstrates that "activation-aware" design is the key to unlocking on-device learning. By breaking the mathematical dependency between weight updates and activation storage, and supplementing the model with lite residual modules, TinyTL enables high-performance model adaptation on memory-constrained edge devices without the traditional overhead of full backpropagation.

Summary updated

Abstract

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 33.8%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.5-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.
BibTeX
@inproceedings{cai2020tinytl,
  author = {Han Cai and Chuang Gan and Ligeng Zhu and Song Han},
  title = {{TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Ckzg9vKidN1},
}
Figure