When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

Michael S. Zhang, Rishi A. Ruia, Arnav Kewalram, Saathvik Dharmapuram, Utkarsh Sharma, Kevin ZhuarXiv 2025 · DOI · arXiv

Abstract

Large Language Models (LLMs) frequently suffer from catastrophic forgetting during sequential fine-tuning, where new knowledge overwrites previously learned information. While quantization is typically viewed as a trade-off between efficiency and accuracy, this study reveals that 8-bit and 4-bit quantization can serve as implicit regularizers that enhance knowledge retention. By evaluating LLaMA-3.1-8B across NLU, Math, and Code tasks, we demonstrate that quantized models often outperform full-precision counterparts in continual learning scenarios, especially when paired with small replay buffers.

Introduction

Fine-tuning Large Language Models (LLMs) on a sequence of tasks often leads to "catastrophic forgetting." To mitigate this, researchers typically employ replay buffers—small subsets of previous data re-introduced during new training phases. However, the interaction between model compression (quantization) and this forgetting process has remained largely unexplored. This paper investigates a counter-intuitive hypothesis: does lowering the precision of a model's weights actually make it better at holding onto old knowledge?

Methodology

The researchers systematically tested the LLaMA-3.1-8B model across three precision levels: FP16 (high-precision baseline), 8-bit (standard weight quantization), and 4-bit (aggressive NF4 quantization). Models were sequentially fine-tuned using LoRA (Low-Rank Adaptation) on three domains: Natural Language Understanding (NLU), Mathematics (GSM8K), and Coding (CodeAlpaca). To evaluate the stability-plasticity trade-off, the study varied the replay buffer size from 0% to 20%.

Quantization as Implicit Regularization

The core insight of this work is that the "noise" introduced by quantization prevents the model from over-optimizing on the gradients of a new task. In FP16, the model has sufficient precision to precisely overwrite old weights to minimize loss on new data. In contrast, the quantized weights are more constrained, which ironically helps anchor existing knowledge and prevents the drastic weight shifts associated with forgetting.

Key Results

The experimental results reveal a striking inversion of performance. While FP16 models typically start with higher initial accuracy, they experience the steepest decline as training progresses through the task sequence.

Retention trends for NLU and Math tasks after subsequent training stages across various replay buffer sizes.

As illustrated in the retention plots, quantized models (INT8 and INT4) consistently maintain higher accuracy on previous tasks compared to the FP16 baseline. Notably, even a minimal replay buffer of 0.1% provides a massive boost in retention across all precision levels, jumping NLU retention significantly.

The most significant finding occurs in the final task performance. On the Code generation task—the final stage in the sequence—the quantized models significantly outperformed the full-precision baseline.

Performance on the final Code task (HumanEval pass@1) across different replay buffer sizes.

As shown in the Code Performance graph, the INT4 and INT8 models achieved accuracy levels near 40% with moderate replay buffers, whereas the FP16 model lagged significantly, often struggling to exceed 25%. This suggests that for complex sequential learning, quantization is not just an efficiency tool but a performance enhancer that requires less replay data to achieve superior stability.

Conclusion

This research challenges the conventional wisdom that quantization necessarily degrades model quality. In the context of continual learning, 8-bit and 4-bit quantization provide a stabilizing effect that mitigates catastrophic forgetting. These findings suggest that future LLM training pipelines could leverage quantization not just for deployment efficiency, but as a core strategy for building more robust, lifelong learning systems.

Paper Citation: Zhang, M. S., et al. (2025). When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models. 39th Conference on Neural Information Processing Systems (NeurIPS 2025).

Summary updated

Abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.
BibTeX
@misc{zhang2025when,
  author = {Michael S. Zhang and Rishi A. Ruia and Arnav Kewalram and Saathvik Dharmapuram and Utkarsh Sharma and Kevin Zhu},
  title = {{When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models}},
  year = {2025},
  eprint = {2512.18934},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.18934},
}
Figure