Introduction
Click-Through Rate (CTR) prediction is a cornerstone of modern digital ecosystems, powering everything from personalized recommendations to online advertising. Traditional models rely on mapping categorical features into dense embedding vectors and modeling their interactions. However, as datasets grow in complexity, these models often suffer from massive parameter bloat and noise from uninformative features. This paper introduces OptFS (Optimizing Feature Set), a framework designed to jointly optimize feature selection and interactions at a granular, feature-level scale.
The Problem: Coarse Selection and Disconnected Optimization
Current CTR models face two primary limitations in feature management. First, most selection methods operate at the "field level" (e.g., keeping or dropping the entire City field). This is often too blunt, as specific values within a field may be highly predictive while others are merely noise. Second, feature selection and interaction selection are typically treated as disjoint tasks. This separation leads to suboptimal configurations where models either overfit on noisy interactions or miss critical patterns due to aggressive pruning.
The OptFS Framework
OptFS addresses these challenges by shifting the optimization focus from fields to individual features. The framework is built on three pillars: feature-level gating, interaction decomposition, and a differentiable training strategy.
Feature-Level Gating and Interaction Decomposition
Instead of gating an entire field, OptFS assigns a learnable binary gate ($g$) to every individual feature value. To manage the combinatorial explosion of potential interactions (which can reach $10^{12}$ pairs), the authors propose an Interaction Decomposition strategy. This approach assumes that an interaction between two features is only valuable if both constituent features are themselves useful. Consequently, the gate for an interaction ($g_{i,j}$) is represented as the product of two individual feature gates ($g_i \times g_j$).
This unification allows the model to search for the optimal feature set and interaction set simultaneously, significantly reducing the search space complexity while maintaining high representational power.
Learning-by-Continuation
Binary gates are non-differentiable, making them difficult to optimize via standard gradient descent. OptFS employs a "learning-by-continuation" approach using a temperature-controlled sigmoid function. During the initial Searching Stage, a low temperature makes the gates "soft," allowing gradients to flow and the model to explore the feature space. As training progresses, the temperature is increased, forcing the gates toward "hard" binary values (0 or 1). Finally, a Retraining Stage fine-tunes the selected features from a stable checkpoint to ensure peak performance.
Experimental Evaluation
The authors evaluated OptFS on three industry-standard benchmarks: Criteo, Avazu, and KDD12. The results demonstrate that OptFS is not only more accurate but also significantly more efficient than existing state-of-the-art methods.
Key Results
- Massive Compression: On the Criteo dataset, OptFS reduced the feature set size by 86% to 96% depending on the backbone model. This reduction directly translates to smaller embedding tables and lower storage requirements.
- Superior Accuracy: Despite the drastic reduction in features, OptFS consistently improved the Area Under Curve (AUC) across all datasets. For example, it achieved an AUC improvement of up to 0.45% over the backbone on Avazu—a significant margin in production CTR environments.
- Inference Efficiency: The streamlined feature set allows for lower inference latency, making OptFS highly suitable for real-time applications.
Conclusion
OptFS demonstrates that in the realm of CTR prediction, "less is more." By moving to a fine-grained, feature-level gating mechanism and unifying it with interaction selection through decomposition, the framework achieves a rare trifecta: higher accuracy, massive parameter reduction, and improved inference speed. Future work may explore extending this logic to higher-order interactions and developing dynamic selection methods that adapt to real-time user contexts.
Summary updated