Optimizing Feature Set for Click-Through Rate Prediction

Fuyuan Lyu, Xing Tang, Dugang Liu, Liang Chen, Xiuqiang He, Xue Liu

Introduction

Click-Through Rate (CTR) prediction is a cornerstone of modern digital ecosystems, powering everything from personalized recommendations to online advertising. Traditional models rely on mapping categorical features into dense embedding vectors and modeling their interactions. However, as datasets grow in complexity, these models often suffer from massive parameter bloat and noise from uninformative features. This paper introduces OptFS (Optimizing Feature Set), a framework designed to jointly optimize feature selection and interactions at a granular, feature-level scale.

The Problem: Coarse Selection and Disconnected Optimization

Current CTR models face two primary limitations in feature management. First, most selection methods operate at the "field level" (e.g., keeping or dropping the entire City field). This is often too blunt, as specific values within a field may be highly predictive while others are merely noise. Second, feature selection and interaction selection are typically treated as disjoint tasks. This separation leads to suboptimal configurations where models either overfit on noisy interactions or miss critical patterns due to aggressive pruning.

The OptFS Framework

OptFS addresses these challenges by shifting the optimization focus from fields to individual features. The framework is built on three pillars: feature-level gating, interaction decomposition, and a differentiable training strategy.

Feature-Level Gating and Interaction Decomposition

Instead of gating an entire field, OptFS assigns a learnable binary gate ($g$) to every individual feature value. To manage the combinatorial explosion of potential interactions (which can reach $10^{12}$ pairs), the authors propose an Interaction Decomposition strategy. This approach assumes that an interaction between two features is only valuable if both constituent features are themselves useful. Consequently, the gate for an interaction ($g_{i,j}$) is represented as the product of two individual feature gates ($g_i \times g_j$).

The Interaction Decomposition mechanism unifies feature and interaction selection into a single searchable space.

This unification allows the model to search for the optimal feature set and interaction set simultaneously, significantly reducing the search space complexity while maintaining high representational power.

Learning-by-Continuation

Binary gates are non-differentiable, making them difficult to optimize via standard gradient descent. OptFS employs a "learning-by-continuation" approach using a temperature-controlled sigmoid function. During the initial Searching Stage, a low temperature makes the gates "soft," allowing gradients to flow and the model to explore the feature space. As training progresses, the temperature is increased, forcing the gates toward "hard" binary values (0 or 1). Finally, a Retraining Stage fine-tunes the selected features from a stable checkpoint to ensure peak performance.

Experimental Evaluation

The authors evaluated OptFS on three industry-standard benchmarks: Criteo, Avazu, and KDD12. The results demonstrate that OptFS is not only more accurate but also significantly more efficient than existing state-of-the-art methods.

Performance comparison of OptFS against backbone models and other selection methods across different datasets.

Key Results

  • Massive Compression: On the Criteo dataset, OptFS reduced the feature set size by 86% to 96% depending on the backbone model. This reduction directly translates to smaller embedding tables and lower storage requirements.
  • Superior Accuracy: Despite the drastic reduction in features, OptFS consistently improved the Area Under Curve (AUC) across all datasets. For example, it achieved an AUC improvement of up to 0.45% over the backbone on Avazu—a significant margin in production CTR environments.
  • Inference Efficiency: The streamlined feature set allows for lower inference latency, making OptFS highly suitable for real-time applications.

Conclusion

OptFS demonstrates that in the realm of CTR prediction, "less is more." By moving to a fine-grained, feature-level gating mechanism and unifying it with interaction selection through decomposition, the framework achieves a rare trifecta: higher accuracy, massive parameter reduction, and improved inference speed. Future work may explore extending this logic to higher-order interactions and developing dynamic selection methods that adapt to real-time user contexts.

Summary updated

Abstract

Click-through prediction (CTR) models transform features into latent vectors and enumerate possible feature interactions to improve performance based on the input feature set. Therefore, when selecting an optimal feature set, we should consider the influence of both feature and its interaction. However, most previous works focus on either feature field selection or only select feature interaction based on the fixed feature set to produce the feature set. The former restricts search space to the feature field, which is too coarse to determine subtle features. They also do not filter useless feature interactions, leading to higher computation costs and degraded model performance. The latter identifies useful feature interaction from all available features, resulting in many redundant features in the feature set. In this paper, we propose a novel method named OptFS to address these problems. To unify the selection of feature and its interaction, we decompose the selection of each feature interaction into the selection of two correlated features. Such a decomposition makes the model end-to-end trainable given various feature interaction operations. By adopting feature-level search space, we set a learnable gate to determine whether each feature should be within the feature set. Because of the large-scale search space, we develop a learning-by-continuation training scheme to learn such gates. Hence, OptFS generates the feature set only containing features which improve the final prediction results. Experimentally, we evaluate OptFS on three public datasets, demonstrating OptFS can optimize feature sets which enhance the model performance and further reduce both the storage and computational cost.
BibTeX
@inproceedings{lyu2023optimizing,
  author = {Fuyuan Lyu and Xing Tang and Dugang Liu and Liang Chen and Xiuqiang He and Xue Liu},
  title = {{Optimizing Feature Set for Click-Through Rate Prediction}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583545},
}
Figure