Abstract
Incremental learning is essential for maintaining the performance of Click-Through Rate (CTR) prediction models in dynamic production environments. However, these models often suffer from "feature staleness," where embeddings for features absent in recent data batches become incompatible with updated high-level model layers. This paper introduces FeSAIL, a framework designed to mitigate this issue through two novel components: Staleness Aware Sampling (SAS) and Staleness Aware Regularization (SAR). SAS employs a greedy algorithm to replay historical data containing relevant stale features, while SAR prevents overfitting on long-term stale features via adaptive regularization. Evaluations on four large-scale datasets show that FeSAIL consistently outperforms existing incremental learning methods, achieving significant gains in AUC and Logloss.
Introduction
In modern recommender systems, models must process billions of interactions daily. To adapt to shifting user preferences and item popularity, models are updated incrementally using the latest data batches. A typical CTR model architecture consists of a low-level embedding layer and high-level interaction layers.
The "Feature Staleness" problem arises when specific features—such as niche product categories or infrequent user IDs—do not appear in the current incremental training batch. While the high-level layers are updated to reflect new trends, the embeddings for these absent features remain stagnant. When a stale feature eventually reappears, its embedding is no longer aligned with the evolved high-level layers, leading to a drop in prediction accuracy (AUC).
The FeSAIL Framework
The proposed FeSAIL (Feature Staleness Aware Incremental Learning) framework introduces a robust pipeline to maintain embedding compatibility during incremental updates.
Staleness Aware Sampling (SAS)
To keep embeddings "in sync," FeSAIL replays a subset of historical data. Since replaying all history is computationally prohibitive, SAS treats sampling as a Maximum Weighted Coverage problem. It utilizes a greedy algorithm to select a fixed-size reservoir of historical samples that cover as many stale features as possible. Crucially, SAS prioritizes "recently stale" features—those that have been absent for a short duration—as they are statistically more likely to reappear soon and require immediate alignment.
Staleness Aware Regularization (SAR)
Not all stale features warrant aggressive updates. Features absent for extended periods are often low-frequency "noise." Updating these embeddings too heavily based on limited replayed samples can lead to overfitting. SAR introduces a "guard" term—a regularization loss that constrains the magnitude of embedding updates based on the feature's staleness. The higher the staleness, the more the update is restricted, ensuring model stability.
Experimental Evaluation
FeSAIL was evaluated against state-of-the-art incremental learning baselines, including ASMG and EWC, across four diverse datasets:
- Criteo & iPinYou: Public advertising datasets with 10M+ and 21M+ samples, respectively.
- Avazu: Mobile ad click data containing 40M+ samples.
- Media: A large-scale industrial dataset from a commercial platform with 104M+ samples.
The experiments simulated a real-world streaming environment by splitting data into a pre-training phase followed by ten consecutive incremental batches.
Results and Discussion
FeSAIL demonstrated superior performance across all metrics and datasets. The framework achieved an average AUC improvement of 1.21% over competing methods. In the context of industrial CTR prediction, improvements of even 0.1% are considered highly significant for platform revenue.
Key findings include:
- Efficiency: FeSAIL maintains high accuracy using a small, fixed-size reservoir, making it suitable for large-scale production systems.
- Robustness: The greedy SAS algorithm provides a theoretical approximation ratio of $1 - 1/e$, ensuring near-optimal feature coverage.
- Disentangled Benefit: Performance gains were specifically observed on samples containing stale features without compromising accuracy on fresh data.
Conclusion
FeSAIL effectively addresses the feature staleness problem in incremental CTR prediction. By combining intelligent historical data replay with staleness-aware regularization, the framework ensures that feature embeddings remain compatible with evolving model layers. Future research will explore more granular time-steps for staleness and the application of these techniques to broader streaming recommendation tasks.
Summary updated