RewardRating: A Mechanism Design Approach to Improve Rating Systems

Iman Vakilinia, Peyman Faizian, Mohammad Mahdi Khalili

Online reviews are the lifeblood of the modern economy. Whether you’re picking a local bistro or a new laptop, you likely check the star rating first. But there’s a problem: the system is easily gamed. From business owners buying 5-star "glow" reviews to malicious competitors tanking rivals with 1-star hits, the "aggregated score" is often a battlefield of misinformation.

In their paper, "RewardRating: A Mechanism Design Approach to Improve Rating Systems," researchers Iman Vakilinia, Peyman Faizian, and Mohammad Mahdi Khalili propose a provocative solution: stop trying to "detect" fakes with AI and start making them too expensive to maintain. Their secret? Turning the rating system into a mini stock market.

TL;DR

RewardRating is a novel mechanism that requires users to "invest" in their ratings using virtual coins. By redistributing the fees from new ratings to existing "shareholders" who predicted the service's quality correctly, the system makes fake ratings exponentially expensive for attackers while rewarding honest, early contributors.

What Problem Does This Paper Address?

Current defenses against fake ratings usually rely on machine learning filters to spot linguistic patterns or suspicious metadata. However, these are cat-and-mouse games; attackers evolve, and ratings are inherently subjective (one person’s 1-star "slow service" is another’s 5-star "relaxed atmosphere").

The authors specifically target Sybil attacks, where a single attacker creates hundreds of fake accounts to swing a score. Since most rating platforms are free to use, the cost of such an attack is near zero. The paper asks: How can we design a system where being honest is profitable and being a liar is a financial sinkhole?

Methods

The authors introduce RewardRating, a mechanism built on the principles of game theory and market dynamics. Here is how it works:

  1. Skin in the Game: To leave a rating, you must buy a "coin" associated with that score (e.g., a 4-star coin).
  2. The Spread: The system sets a buying price ($\alpha$) and a lower selling price ($\beta$). The difference ($\gamma$) is the "profit" or fee generated by the transaction.
  3. Profit Sharing: This fee isn't just pocketed by the platform. It is distributed among existing stakeholders.
  4. If a new rating is higher than the current average, the fee is shared among those who previously gave high ratings.
  5. If it’s lower, it goes to those who gave low ratings.
  6. Proximity Rewards: The closer your original rating is to the new incoming rating, the larger your share of the profit. The authors propose two mathematical functions ($f_1$ and $f_2$) to calculate this "distance-based" reward.

By doing this, the system becomes budget-balanced (self-sustaining) and incentive-compatible (strategic players maximize utility by being honest).

Datasets

The paper does not use a large-scale real-world dataset. Instead, it utilizes numerical simulations and mathematical modeling to prove its propositions. The authors simulate a restaurant scenario with varying numbers of "honest coins" (100, 200, and 500) to analyze how an attacker's costs scale when they try to manipulate the aggregated score.

Key Insights and Ideas

  • The Exponential Wall: The most striking insight is that the cost to manipulate a score doesn't grow linearly—it grows exponentially. To move a 1-star average to a 4-star average in a popular system, an attacker has to buy an increasingly massive number of coins to "dilute" the honest pool.
  • Attackers Fund the Honest: In this model, when a malicious owner buys fake 5-star reviews, the fees they pay are actually distributed to the honest users who already gave the business high marks.
  • Strategic Honesty: For a "strategic" user (someone just looking to make money), the best move is to predict what the final stable rating will be. If a service is truly good, betting on a high rating early is the most profitable strategy, aligning the profit motive with the truth.

Key Results

  • Budget Balance: The authors mathematically prove that the system is self-sustaining; the total assets put in by users equal the total output (rewards + selling price).
  • Incentive Compatibility: The paper sketches a proof showing that honest users profit from future honest participation, while attackers gain no profit from honest users because honest users will naturally "bet" in the opposite direction of a fake pump-and-dump.
  • Cost Analysis: Simulation results show that as the number of honest reviewers increases, the financial barrier for an attacker becomes insurmountable. For a service with 500 honest coins, the cost for an attacker to significantly move the needle spikes vertically (see Figure 6 in the paper).

Limitations and Future Work

The authors are transparent about the hurdles:

  • The Barrier to Entry: Requiring people to pay to rate things is a massive shift in user behavior. Most users expect rating systems to be free.
  • Profit Dilution: As more people join the "stakeholder" pool for a specific rating, the individual payout from new ratings gets smaller, which might discourage new honest users from participating.
  • Implementation: The authors suggest that the future of this system lies in blockchain and smart contracts. They propose a browser add-on where users could use cryptocurrency to "invest" in ratings, removing the need for a trusted third party to manage the funds.

By moving the problem from the realm of "pattern recognition" to "economic equilibrium," RewardRating offers a fascinating, if controversial, blueprint for a more trustworthy internet.

Summary updated

Abstract

Nowadays, rating systems play a crucial role in the attraction of customers for different services. However, as it is difficult to detect a fake rating, attackers can potentially impact the rating's aggregated score unfairly. This malicious behavior can negatively affect users and businesses. To overcome this problem, we take a mechanism-design approach to increase the cost of fake ratings while providing incentives for honest ratings. Our proposed mechanism \textit{RewardRating} is inspired by the stock market model in which users can invest in their ratings for services and receive a reward based on future ratings. First, we formally model the problem and discuss budget-balanced and incentive-compatibility specifications. Then, we suggest a profit-sharing scheme to cover the rating system's requirements. Finally, we analyze the performance of our proposed mechanism.
BibTeX
@inproceedings{vakilinia2022rewardrating,
  author = {Iman Vakilinia and Peyman Faizian and Mohammad Mahdi Khalili},
  title = {{RewardRating: A Mechanism Design Approach to Improve Rating Systems}},
  booktitle = {Games},
  year = {2022},
  url = {https://openreview.net/forum?id=2gl9JAnH25},
}

Loading related content...

Figure