Overcoming Forgetting Catastrophe in Quantization-Aware Training

Ting-An Chen, De-Nian Yang, Ming-Syan ChenICCV 2023 · OpenReview

Abstract

Quantization is an effective approach for memory cost reduction by compressing networks to lower bits. However, existing quantization processes learned only from the current data tend to suffer from forgetting catastrophe on streaming data, i.e., significant performance decrement on old task data after being trained on new tasks. Therefore, we propose a lifelong quantization process, LifeQuant, to address the problem. We theoretically analyze the forgetting catastrophe from the shift of quantization search space with the change of data tasks. To overcome the forgetting catastrophe, we first minimize the space shift during quantization and propose Proximal Quantization Space Search (ProxQ), for regularizing the search space during quantization to be close to a pre-defined standard space. Afterward, we exploit replay data (a subset of old task data) for retraining in new tasks to alleviate the forgetting problem. However, the limited amount of replay data usually leads to biased quantization performance toward the new tasks. To address the imbalance issue, we design a Balanced Lifelong Learning (BaLL) Loss to reweight (to increase) the influence of replay data in new task learning, by leveraging the class distributions. Experimental results show that LifeQuant achieves outstanding accuracy performance with a low forgetting rate.

Introduction

Deep neural networks are often too large for deployment on edge devices like smartphones and IoT sensors, necessitating compression techniques like Quantization-Aware Training (QAT). While QAT effectively reduces memory costs by using low-bit formats, it typically assumes a static dataset. In real-world scenarios, data often arrives in sequential streams. When a model is quantized for a new task, it frequently suffers from catastrophic forgetting—a massive performance drop on previously learned tasks.

The authors of LifeQuant identify that this forgetting is primarily caused by a "space shift" in the quantization search space. As the model adapts to new data, the range and distribution of values it uses to represent weights shift drastically, making it impossible to accurately represent knowledge from previous tasks using the new discrete levels.

Methodology: The LifeQuant Framework

Proximal Quantization Space Search (ProxQ)

The authors theoretically prove (Theorem 5.1) that the increase in quantization error is directly linked to the shift in quantization parameters between tasks. In standard quantization, the search space evolves freely, leading to significant forgetting as the representation for old weights is lost.

ProxQ regularizes the quantization search space to stay close to a pre-defined standard space, minimizing shift between tasks.

ProxQ addresses this by forcing the quantization search space of every task to remain close to a pre-defined "standard" Gaussian space. By anchoring these discrete levels, the model preserves its ability to represent old weights even as it adapts to new data distributions.

Balanced Lifelong Learning (BaLL) Loss

To further mitigate forgetting, LifeQuant utilizes "replay data"—a small subset of samples from previous tasks. However, the extreme imbalance between new task data and the tiny replay buffer often biases the model toward the current task.

BaLL Loss uses class distribution statistics to reweight the influence of replay data, ensuring balanced performance.

The BaLL Loss function reweights the influence of replay data based on class distributions. By "boosting" the signal from old tasks, it ensures the model remains balanced, a factor that is particularly critical in low-bit regimes (like 2-bit) where quantization error amplifies the negative effects of data imbalance.

Experimental Results

LifeQuant was evaluated across benchmarks including CIFAR-100, Office-31, and ImageCLEF. The results demonstrate significant improvements over existing state-of-the-art QAT methods:

  • Accuracy Gains: On ImageCLEF using MobileNet-V2 (3-bit), LifeQuant achieved a 17% accuracy improvement.
  • Reduced Forgetting: The forgetting rate was reduced by 23% on the same dataset.
  • Ablation Analysis: Removing the ProxQ component led to a 30% accuracy drop on ImageCLEF, highlighting the critical importance of managing the quantization search space shift.

Conclusion

LifeQuant successfully addresses catastrophic forgetting in quantization-aware training by combining search space regularization with class-balanced learning. While the reliance on a replay buffer remains a minor limitation for extremely memory-constrained or privacy-sensitive environments, the framework sets a new standard for lifelong learning in compressed neural networks. Future work may explore non-uniform quantization and continuous task streams where boundaries are not explicitly defined.

Summary updated

Abstract

Quantization is an effective approach for memory cost reduction by compressing networks to lower bits. However, existing quantization processes learned only from the current data tend to suffer from forgetting catastrophe on streaming data, i.e., significant performance decrement on old task data after being trained on new tasks. Therefore, we propose a lifelong quantization process, LifeQuant, to address the problem. We theoretically analyze the forgetting catastrophe from the shift of quantization search space with the change of data tasks. To overcome the forgetting catastrophe, we first minimize the space shift during quantization and propose Proximal Quantization Space Search (ProxQ), for regularizing the search space during quantization to be close to a pre-defined standard space. Afterward, we exploit replay data (a subset of old task data) for retraining in new tasks to alleviate the forgetting problem. However, the limited amount of replay data usually leads to biased quantization performance toward the new tasks. To address the imbalance issue, we design a Balanced Lifelong Learning (BaLL) Loss to reweight (to increase) the influence of replay data in new task learning, by leveraging the class distributions. Experimental results show that LifeQuant achieves outstanding accuracy performance with a low forgetting rate.
BibTeX
@inproceedings{chen2023overcoming,
  author = {Ting-An Chen and De-Nian Yang and Ming-Syan Chen},
  title = {{Overcoming Forgetting Catastrophe in Quantization-Aware Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=wqz6K6BFBg},
}
Figure