Recommendation Algorithms for Optimizing Hit Rate, User Satisfaction and Website Revenue

Xin Wang, Yunhui Guo, Congfu XuIJCAI 2015

Abstract

Traditional recommender systems prioritize "hit rate"—the likelihood of a click or purchase—often at the expense of long-term user satisfaction and retailer profitability. This paper presents HSR-1 and HSR-2, two novel algorithms that frame recommendation as a multi-objective optimization problem. By integrating purchase probability, user ratings, and item prices into a unified matrix factorization framework, these models identify items that are not only likely to be purchased but also highly satisfying for the user and profitable for the retailer.

Introduction

Most recommendation research focuses on either predicting ratings or ranking items to maximize clicks. However, a high "hit rate" does not necessarily translate to a successful business outcome. Two critical gaps exist in current models:

  1. User Satisfaction: Recommending items that lead to "buyer's remorse" erodes long-term trust, even if the initial purchase was a "hit."
  2. Website Revenue: If multiple items have similar purchase probabilities, systems should prioritize those with higher profit margins.

The proposed HSR (Hit rate, Satisfaction, and Revenue) models address these gaps by predicting "satisfying, profitable hits."

Methodology

The authors introduce two variants of the HSR model, both extending Matrix Factorization (MF) to handle multi-objective optimization.

HSR-1: Bivariate Normal Approach

HSR-1 employs a Bernoulli distribution to model the binary "hit" (purchase) event. To incorporate satisfaction and revenue, it assumes that user ratings and item prices follow a bivariate normal distribution. This allows the model to capture the inherent correlation between user preference and price points. Latent factors for users and items are learned via Stochastic Gradient Descent (SGD).

HSR-2: Binomial Robustness

Recognizing that star ratings are discrete (1-5), HSR-2 replaces the Gaussian assumption with a Binomial distribution. This approach is more mathematically robust for discrete ordinal data and prevents optimization from being constrained by the artificial bounds of a continuous rating scale.

Both models utilize price not just as a target but as a feature to learn a user's "acceptable price range," which significantly refines hit rate predictions.

Experimental Setup

The models were evaluated using 10 real-world Amazon datasets spanning categories such as Automotive, Beauty, and Clothing. These datasets provide the necessary signals for multi-objective evaluation: purchase history, star ratings, and item prices (serving as a proxy for revenue).

Figure 1: Distribution of ratings across various Amazon product categories, showing a high proportion of 4 and 5-star reviews.

Data was pre-processed to ensure complete price and rating information, using an 80/10/10 split for training, validation, and testing.

Key Insights and Results

The core innovation of HSR is the shift from "Direction A" (optimizing for user preference alone) to "Direction B" (optimizing for the intersection of user satisfaction and retailer revenue).

Figure 4: Comparison of optimization strategies: Traditional models (Direction A) vs. HSR models (Direction B) targeting the intersection of user and retailer satisfaction.

Performance Gains

  • Superior Hit Rate: HSR models outperformed state-of-the-art baselines like BPR and iMF. F1 scores improved by 7.7% to 36.2% across datasets, proving that multi-objective optimization does not have to come at the cost of accuracy.
  • Revenue and Satisfaction: Using new metrics like Satisfaction Recall (S-R) and Price Recall (P-R), the authors demonstrated massive gains. In categories like "Industrial & Scientific," revenue-related performance nearly doubled.
  • Price as a Signal: Including price information improved hit rate by helping the model avoid recommending items outside a user's perceived value or budget range.

Conclusion

HSR-1 and HSR-2 demonstrate that incorporating business logic and user satisfaction into recommendation engines leads to better outcomes for both users and retailers. While the current work uses price as a proxy for revenue, future iterations could incorporate actual profit margins and explore recommendation diversity to further enhance the user experience.

Summary updated

BibTeX
@inproceedings{wang2015recommendation,
  author = {Xin Wang and Yunhui Guo and Congfu Xu},
  title = {{Recommendation Algorithms for Optimizing Hit Rate, User Satisfaction and Website Revenue}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure