Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning

Han Cai, Chuang Gan, Ligeng Zhu, Song Han

Abstract

On-device learning enables edge devices to continually adapt AI models to new data, requiring a small memory footprint to fit tight constraints. Existing work often reduces trainable parameters, but this doesn't directly translate to memory savings since the major bottleneck is activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learning the bias modules, thus eliminating the need to store intermediate activations. To maintain adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps, adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.

The Activation Bottleneck in On-Device Learning

As AI moves to edge devices like IoT sensors and smartphones, on-device learning becomes crucial for privacy and local adaptation. However, training is limited by extreme memory constraints. A common misconception in the field is that parameter-efficient models (e.g., MobileNetV2) are inherently training-memory-efficient. In reality, the activations—intermediate feature maps stored during the forward pass to calculate gradients during the backward pass—are the primary bottleneck. For instance, while MobileNetV2 has 4.3x fewer parameters than ResNet-50, their training activation footprints are nearly identical. To make on-device learning viable, we must reduce the activation footprint, not just the model size.

Tiny-Transfer-Learning (TinyTL)

To address this, TinyTL shifts the focus from parameter reduction to activation reduction through three main strategies:

  1. Weight Freezing & Bias-Only Tuning: Gradients for weights require storing input activations, but gradients for biases do not. By freezing weights and only updating biases, TinyTL avoids storing almost all intermediate activations.
  2. Lite Residual Module: To restore the adaptation capacity lost by freezing weights, TinyTL adds a small, trainable branch. This module uses reduced resolution (via 2x2 average pooling) and reduced width (group convolutions) to refine features with only ~3.8% additional memory overhead.
  3. Feature Extractor Adaptation: TinyTL uses a Once-for-All (OFA) network to select a specialized sub-network backbone that best fits the target dataset before the fine-tuning process begins.
The TinyTL architecture compared to conventional fine-tuning. (a) Conventional full fine-tuning stores all activations. (b) Bias-only tuning avoids activation storage but loses capacity. (c) TinyTL's lite residual learning restores capacity with minimal memory overhead.

Experimental Evaluation

TinyTL was evaluated across a wide array of computer vision benchmarks, including Stanford Cars, Flowers-102, CUB-200, and CIFAR-100.

Key Results

  • Massive Memory Savings: TinyTL provides 7.3x to 12.9x memory savings compared to fine-tuning a full Inception-V3 model without sacrificing accuracy.
  • Superior Accuracy: Compared to the standard practice of only fine-tuning the "last layer," TinyTL improves accuracy by massive margins—up to 34.1% on the Stanford Cars dataset.
  • SRAM-Ready Training: By reducing the memory footprint to as low as 16MB (at batch size 1), TinyTL allows the training state to fit within a processor's L3 cache (SRAM). This avoids slow and power-hungry DRAM access, potentially making on-device training significantly faster and more energy-efficient.
Accuracy vs. Training Memory across various datasets, showing TinyTL's (red) significant memory savings and superior accuracy compared to full fine-tuning (yellow) and other baselines.

Conclusion and Limitations

TinyTL demonstrates that focusing on activation reduction is the key to viable on-device learning. While highly effective for transfer learning, the method is specifically designed for adapting pre-trained models rather than training from scratch. It also benefits from Group Normalization over Batch Normalization to maintain efficiency at small batch sizes. Future work may extend these principles to activation-heavy architectures in other domains, such as Natural Language Processing.

Summary updated

Abstract

On-device learning enables edge devices to continually adapt the AI models to new data, which requires a small memory footprint to fit the tight memory constraint of edge devices. Existing work solves this problem by reducing the number of trainable parameters. However, this doesn't directly translate to memory saving since the major bottleneck is the activations, not parameters. In this work, we present Tiny-Transfer-Learning (TinyTL) for memory-efficient on-device learning. TinyTL freezes the weights while only learns the bias modules, thus no need to store the intermediate activations. To maintain the adaptation capacity, we introduce a new memory-efficient bias module, the lite residual module, to refine the feature extractor by learning small residual feature maps adding only 3.8% memory overhead. Extensive experiments show that TinyTL significantly saves the memory (up to 6.5x) with little accuracy loss compared to fine-tuning the full network. Compared to fine-tuning the last layer, TinyTL provides significant accuracy improvements (up to 34.1%) with little memory overhead. Furthermore, combined with feature extractor adaptation, TinyTL provides 7.3-12.9x memory saving without sacrificing accuracy compared to fine-tuning the full Inception-V3.
BibTeX
@inproceedings{cai2020tiny,
  author = {Han Cai and Chuang Gan and Ligeng Zhu and Song Han},
  title = {{Tiny Transfer Learning: Towards Memory-Efficient On-Device Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Ckzg9vKidN1},
}

Loading related content...

Figure