Abstract
In recommendation systems, Click-Through Rate (CTR) models must be updated incrementally to capture evolving user interests. However, standard incremental learning often leads to "feature staleness," where embeddings for infrequent features become incompatible with the rapidly updating high-level interaction layers. This paper introduces FeSAIL, a framework designed to mitigate this issue through a Staleness Aware Sampling (SAS) strategy and a Staleness Aware Regularization (SAR) mechanism. Experimental results across four large-scale datasets demonstrate that FeSAIL significantly outperforms existing state-of-the-art incremental learning methods, achieving an average AUC improvement of 1.21%.
Introduction
Modern recommendation systems process billions of interactions daily, necessitating Incremental Learning (IL) to fine-tune models on new data batches. CTR models typically comprise a low-level embedding layer and high-level interaction layers. A critical challenge arises when specific features do not appear in a recent data batch: their embeddings remain static while the rest of the model evolves. When these "stale" features eventually reappear, their embeddings are no longer synchronized with the updated interaction layers, leading to degraded predictive performance. This "feature staleness problem" is not easily solved by random sampling or full retraining due to computational constraints.
Methodology: The FeSAIL Framework
The FeSAIL (Feature Staleness Aware Incremental Learning) framework addresses the synchronization gap through two primary components: data sampling and model training.
Staleness Aware Sampling (SAS)
Instead of relying on random historical data, SAS maintains a fixed-size reservoir and selects samples that maximize the coverage of stale features. The selection is modeled as a "Maximum Weighted Coverage" problem, prioritizing features with lower staleness values—those that have been absent for a short duration. These features are prioritized because they are more likely to reappear in the near future and require immediate synchronization. To ensure production-level efficiency, the authors implemented a neighbor-based optimization that restricts the search space to samples sharing specific stale features.
Staleness Aware Regularization (SAR)
To prevent overfitting on features that have been stale for extended periods, SAR introduces a "guard" mechanism. This regularization term restricts the magnitude of embedding updates based on the feature's specific staleness level. By normalizing staleness across the dataset, SAR provides fine-grained control, ensuring that updates are meaningful rather than erratic jumps for long-unseen features.
Experiments and Results
The researchers evaluated FeSAIL against several baselines, including standard Incremental Update (IU), Random Sampling (RS), and specialized IL methods like Elastic Weight Consolidation (EWC). Testing was conducted on four diverse datasets: Criteo, iPinYou, Avazu, and a private industrial dataset (Media).
FeSAIL consistently achieved the highest AUC and lowest Logloss across all time spans and datasets. Notably, it delivered an average 1.21% AUC improvement over existing state-of-the-art methods. In the context of industrial CTR prediction, where gains of 0.1% are considered substantial, this represents a significant advancement. Ablation studies further confirmed that both SAS and SAR are critical to these performance gains; removing either component resulted in a measurable drop in accuracy.
Conclusion
FeSAIL effectively addresses the feature staleness problem in incremental CTR prediction by combining intelligent historical data replay with staleness-aware regularization. The framework is model-agnostic and demonstrates superior efficiency and accuracy compared to traditional sampling and regularization techniques. Future work includes extending FeSAIL to more complex architectures like Transformers and automating hyperparameter tuning for the staleness threshold and regularization coefficients.
Summary updated