Introduction
Sequential Recommendation Systems (SRecsys) are foundational to modern digital experiences, yet they often struggle with a critical aspect of user interaction: uncertainty. Traditional models are typically trained using Cross-Entropy (CE) loss, which focuses on maximizing the probability of a single ground-truth item. This approach leads to "confidence blindness," where the model cannot distinguish between high-certainty and low-certainty predictions. Furthermore, CE loss is misaligned with the practical "Top-K" nature of recommendations, treating the task as a winner-takes-all classification rather than a set-based retrieval problem.
The CPFT Framework
To address these issues, researchers have introduced CPFT (Confidence-aware Fine-Tuning). This framework integrates Conformal Prediction (CP)—a statistical method that provides mathematical coverage guarantees—directly into the model's training loop. Unlike post-hoc CP applications, CPFT uses differentiable proxy losses to optimize the model's internal representations for both accuracy and prediction efficiency.
The methodology involves a two-stage process. First, a model (such as SASRec or BERT4Rec) undergoes standard training. In the second stage, it is fine-tuned using a combination of standard prediction loss and two novel CP-based losses:
- Conformal Prediction Set Size (CPS) Loss: This loss penalizes large, inefficient prediction sets, encouraging the model to be more decisive and produce smaller, high-confidence lists.
- Conformal Prediction Set Distance (CPD) Loss: This loss uses embedding similarity to pull the top-ranked items in the prediction set closer to the ground truth, refining the quality of the candidates.
Experimental Setup and Results
CPFT was evaluated on five real-world Amazon Review datasets (Scientific, Pantry, Instruments, Arts, and Office). The results demonstrate that CPFT significantly outperforms standard training across multiple metrics.
Key findings include:
- Accuracy Gains: An average improvement of 4.55% in NDCG and Hit Rate, with particularly strong gains in Recall@50.
- Efficiency: Fine-tuning with CPFT is highly efficient, requiring only 12.27% of the time needed for full training on the "Office" dataset.
- Set Reduction: The framework successfully shrinks prediction sets (e.g., from ~300 to ~25 items) while maintaining the desired statistical coverage, effectively reducing the cognitive load on users.
- Calibration: Qualitative analysis showed that CPFT-refined models assigned significantly higher confidence scores to relevant items (e.g., increasing from 0.0058 to 0.0466 for a ground-truth item).
Conclusion
CPFT represents a significant step toward more trustworthy and efficient recommendation systems. By leveraging Conformal Prediction as a regularizer during fine-tuning, the framework produces models that are not only more accurate but also better calibrated to their own uncertainty. Future work may explore non-exchangeable CP methods to better handle the temporal dependencies inherent in sequential data.
Summary updated