Neural Network Retraining for Model Serving

Diego Klabjan, Xiaofeng ZhuarXiv 2020 · arXiv

Abstract

Maintaining the performance of neural networks in production environments is challenging due to concept drift. Traditional retraining methods often suffer from high computational costs or catastrophic forgetting. This paper presents a novel framework for continuous neural network retraining that integrates neuron importance regularization with Multi-Armed Bandit (MAB) algorithms. By intelligently selecting training data and weight clusters for updates, the proposed method achieves superior accuracy and generalization while remaining computationally efficient for real-time model serving.

Introduction

In real-world model serving, data distributions are rarely static. Models trained on historical data frequently experience performance degradation as underlying patterns shift—a phenomenon known as concept drift. Engineers typically face a trade-off between retraining on the entire accumulated dataset, which is computationally prohibitive, or fine-tuning on only new data, which often leads to catastrophic forgetting of previously learned edge cases. This work introduces a middle ground: an incremental retraining strategy that is both fast enough for production and robust enough to preserve historical knowledge.

Methodology

The proposed framework optimizes the retraining process through three key innovations:

  1. Synapse and Neuron Importance Regularization: Unlike traditional methods that only penalize changes to critical weights, this approach introduces a loss term that penalizes significant shifts in neuron activity. This ensures the model's structural behavior remains consistent, effectively mitigating forgetting.
  2. MAB-based Memory Replay: To optimize the use of historical data, the framework employs a Multi-Armed Bandit (MAB) algorithm. Instead of random sampling, the MAB treats data mini-batches as "arms" and learns to select those that provide the highest "reward," defined by the decrease in loss or gradient norm.
  3. MAB-based Weight Optimization: Observing that many weights in a network are highly correlated, the authors cluster these weights. A second MAB then decides which cluster to update at each training step. This reduces the number of active parameters per iteration, acting as a form of regularization that improves generalization.

Analysis of Weight Dynamics

A core motivation for the weight clustering approach is the observation that weights within different layers often move in tandem during training.

Correlation of weight pairs across different layers (CNN and FC) over training epochs.

As shown in Figure 1, many weight pairs maintain high positive or negative correlations throughout the training process. By identifying and optimizing these correlated weights as single units, the retraining process becomes more efficient and less prone to overfitting.

Experimental Results

The framework was evaluated across six benchmarks, including image classification (MNIST, CIFAR-10), concept drift datasets (SEA, ELEC), and text classification (IMDB, REUTERS).

The MAB-based approach consistently outperformed state-of-the-art reservoir sampling, yielding an average accuracy boost of 0.48%. Notably, the "Clustering + MAB" strategy significantly improved generalization on augmented or distorted test data by an average of 4.9% compared to standard Batch Normalization.

Evolution of weight values and partial derivatives over training epochs.

In terms of efficiency, the MAB-driven selection process allows for rapid updates. For instance, on the REUTERS dataset, the proposed method was nearly 20 times faster than retraining on the full union of old and new data, while maintaining comparable or better performance.

Conclusion and Future Work

This paper demonstrates that combining neuron-level regularization with active resource allocation via MABs provides a powerful solution for life-long model retraining. While the current implementation achieves significant accuracy gains, further performance improvements could be realized through custom sparse tensor kernels optimized for the framework's selective weight updates. Future research will also explore integrating this method with automated drift detection systems and extending it to handle class-incremental learning scenarios.

Summary updated

Abstract

We propose incremental (re)training of a neural network model to cope with a continuous flow of new data in inference during model serving. As such, this is a life-long learning process. We address two challenges of life-long retraining: catastrophic forgetting and efficient retraining. If we combine all past and new data it can easily become intractable to retrain the neural network model. On the other hand, if the model is retrained using only new data, it can easily suffer catastrophic forgetting and thus it is paramount to strike the right balance. Moreover, if we retrain all weights of the model every time new data is collected, retraining tends to require too many computing resources. To solve these two issues, we propose a novel retraining model that can select important samples and important weights utilizing multi-armed bandits. To further address forgetting, we propose a new regularization term focusing on synapse and neuron importance. We analyze multiple datasets to document the outcome of the proposed retraining methods. Various experiments demonstrate that our retraining methodologies mitigate the catastrophic forgetting problem while boosting model performance.
BibTeX
@misc{klabjan2020neural,
  author = {Diego Klabjan and Xiaofeng Zhu},
  title = {{Neural Network Retraining for Model Serving}},
  year = {2020},
  eprint = {2004.14203},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.14203},
}
Figure