Abstract
Large Language Models (LLMs) frequently suffer from catastrophic forgetting during sequential fine-tuning, where new knowledge overwrites previously learned information. While quantization is typically viewed as a trade-off between efficiency and accuracy, this study reveals that 8-bit and 4-bit quantization can serve as implicit regularizers that enhance knowledge retention. By evaluating LLaMA-3.1-8B across NLU, Math, and Code tasks, we demonstrate that quantized models often outperform full-precision counterparts in continual learning scenarios, especially when paired with small replay buffers.
Introduction
Fine-tuning Large Language Models (LLMs) on a sequence of tasks often leads to "catastrophic forgetting." To mitigate this, researchers typically employ replay buffers—small subsets of previous data re-introduced during new training phases. However, the interaction between model compression (quantization) and this forgetting process has remained largely unexplored. This paper investigates a counter-intuitive hypothesis: does lowering the precision of a model's weights actually make it better at holding onto old knowledge?
Methodology
The researchers systematically tested the LLaMA-3.1-8B model across three precision levels: FP16 (high-precision baseline), 8-bit (standard weight quantization), and 4-bit (aggressive NF4 quantization). Models were sequentially fine-tuned using LoRA (Low-Rank Adaptation) on three domains: Natural Language Understanding (NLU), Mathematics (GSM8K), and Coding (CodeAlpaca). To evaluate the stability-plasticity trade-off, the study varied the replay buffer size from 0% to 20%.
Quantization as Implicit Regularization
The core insight of this work is that the "noise" introduced by quantization prevents the model from over-optimizing on the gradients of a new task. In FP16, the model has sufficient precision to precisely overwrite old weights to minimize loss on new data. In contrast, the quantized weights are more constrained, which ironically helps anchor existing knowledge and prevents the drastic weight shifts associated with forgetting.
Key Results
The experimental results reveal a striking inversion of performance. While FP16 models typically start with higher initial accuracy, they experience the steepest decline as training progresses through the task sequence.
As illustrated in the retention plots, quantized models (INT8 and INT4) consistently maintain higher accuracy on previous tasks compared to the FP16 baseline. Notably, even a minimal replay buffer of 0.1% provides a massive boost in retention across all precision levels, jumping NLU retention significantly.
The most significant finding occurs in the final task performance. On the Code generation task—the final stage in the sequence—the quantized models significantly outperformed the full-precision baseline.
As shown in the Code Performance graph, the INT4 and INT8 models achieved accuracy levels near 40% with moderate replay buffers, whereas the FP16 model lagged significantly, often struggling to exceed 25%. This suggests that for complex sequential learning, quantization is not just an efficiency tool but a performance enhancer that requires less replay data to achieve superior stability.
Conclusion
This research challenges the conventional wisdom that quantization necessarily degrades model quality. In the context of continual learning, 8-bit and 4-bit quantization provide a stabilizing effect that mitigates catastrophic forgetting. These findings suggest that future LLM training pipelines could leverage quantization not just for deployment efficiency, but as a core strategy for building more robust, lifelong learning systems.
Paper Citation: Zhang, M. S., et al. (2025). When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models. 39th Conference on Neural Information Processing Systems (NeurIPS 2025).
Summary updated