Abstract: Modern online platforms rely on effective rating systems to learn about items. We consider the optimal design of rating systems that collect binary feedback after transactions. We make three contributions: (1) we formalize the design of binary rating systems as an information maximization problem; (2) we provide an efficient algorithm to find the optimal rating function; and (3) we demonstrate how to implement this in practice by engineering the questions asked to users.
Introduction
Digital marketplaces often struggle with "uninformative" ratings. Whether it's the 5-star inflation on Airbnb or the binary "thumbs up" on Netflix, current systems frequently fail to distinguish between good and great items quickly. When the vast majority of reviews are a perfect score, the platform learns very little about relative quality. This paper, presented at AISTATS 2019, provides a mathematical framework to design binary rating systems that learn the true ranking of items as fast as possible.
The Optimal Rating Function
The core of the model is the rating function $\beta(\theta)$, which represents the probability that an item of true quality $\theta$ receives a positive rating. The authors show that the "optimal" $\beta$ curve is not necessarily a simple linear function. Instead, it depends heavily on the platform's specific objectives. For instance, a platform aiming to identify the top 10% of performers will require a different $\beta$ curve than one focused on filtering out the bottom 10%.
One key insight is that "variance is the enemy." Binomial variance is highest when the probability is 0.5. The optimal system actually "pushes" items away from this high-variance middle ground to make them easier to distinguish, effectively equalizing the "distinguishability" between neighboring quality tiers.
Algorithmic Solution
To find the optimal $\beta$, the authors use Large Deviations Theory to focus on the rate of convergence—the speed at which ranking errors disappear as more ratings are collected. They developed the NestedBisection algorithm, which solves a system of convex equations to find optimal rating levels. The algorithm runs in $O(M \log^2 M)$ time, where $M$ is the number of quality tiers, making it nearly instantaneous for practical applications with hundreds of discrete quality levels.
Empirical Validation and Results
Since platforms cannot directly set a user's internal probability of giving a "thumbs up," the authors propose "question engineering." By asking a variety of binary questions (e.g., "Was this okay?" vs. "Was this phenomenal?"), a platform can approximate the theoretically optimal $\beta(\theta)$ curve.
The authors validated this using TOEFL writing samples and Amazon Mechanical Turk (MTurk). Human raters evaluated paragraphs using different adjectives, allowing the researchers to map out how different questions shift the probability of a positive rating across quality levels.
The results demonstrate that an optimized distribution of questions significantly outperforms naive benchmarks. In both simulations and empirical tests, the optimized system achieved higher Kendall’s $\tau$ correlation with true rankings in fewer steps. Interestingly, the study suggests that "inflation" can actually be optimal if the goal is purely to separate the very worst items from the rest of the pack.
Conclusion and Future Work
While the current model assumes non-adaptive questions and honest user feedback, it provides a robust foundation for more informative rating systems. By moving away from vague "thumbs up" systems toward strategically engineered questions, platforms can extract significantly more information per transaction. Future work could extend this logic to multi-point scales or adaptive questioning strategies to further accelerate the learning process in the digital economy.
Summary updated