Utilising Crowdsourcing to Assess the Effectiveness of Item-based Explanations of Merchant Recommendations

Denis Krasilnikov, Oleg Lashinin, Maksim Tsygankov, Marina Ananyeva, Sergey KolesnikovCSW@WSDM 2023 · OpenReview

Introduction

In modern financial applications, merchant recommendations—such as cashback offers in banking apps—are increasingly common. To foster user trust, these systems often provide item-based explanations such as, "Because you shopped at Merchant X, we recommend Merchant Y." However, a significant challenge remains: mathematical correlations identified by algorithms do not always align with human logic. For instance, an algorithm might link a bar to a children's clothing store based on co-occurrence data, but a user may find this explanation nonsensical. This paper addresses the lack of standardized methods to evaluate whether these explanation pairs pass a "common sense" test.

Methodology

The researchers proposed a novel framework to bridge the gap between algorithmic relevance and human perception. The approach consists of two primary stages:

  1. Crowdsourced Validation: An internal platform was used to collect human judgments on 23,000 unique (X, Y) merchant pairs. Approximately 400 crowdworkers evaluated these pairs by answering whether a recommendation for Merchant Y based on a past purchase at Merchant X seemed correct. Each pair received five independent ratings to establish a consensus ground truth.
  2. Offline Evaluation Framework: Using the crowdsourced labels as a benchmark, the authors evaluated nine different ranking heuristics and models. This allowed for the calculation of standard information retrieval metrics, such as Recall@K and NDCG, to determine which algorithms best mimic human logical reasoning.
Proposed framework for evaluating item-based recommendations using crowdsourced validation.

The models tested ranged from simple baselines like MostPop (global popularity) and Chrono (chronological order) to sophisticated latent factor models like Implicit ALS (iALS) and EASE.

Dataset

The study utilized the TTRS (Tinkoff Transaction Recommender System) dataset, a large-scale open-source collection of retail transactions. To ensure data quality and focus on impactful merchants, the researchers preprocessed the data to include the top 350 most popular merchants, which represent approximately 87% of total transactions. The resulting dataset comprised 88,500 users and 10 million transactions. From this, a subset of 23,000 merchant pairs was generated for human labeling, creating a unique benchmark for explainability.

Key Results

The evaluation revealed a significant "common sense" gap in current recommendation systems. Even the top-performing model, Implicit ALS (iALS), achieved a Recall@1 of only 0.37, indicating that the primary explanation was deemed illogical by humans over 60% of the time.

Performance comparison of various recommendation and ranking heuristics against human-labeled ground truth.

As shown in the results table, iALS consistently outperformed other methods across all metrics (e.g., Recall@10 = 0.91). SimCatIALS, which incorporates category-level logic, followed closely but did not surpass pure iALS, suggesting that rigid category constraints may sometimes overlook subtle logical connections identified by latent factor models. Conversely, popularity-based baselines like MostPop performed poorly, confirming that global popularity is a weak proxy for personalized logical relevance.

Discussion and Limitations

The findings highlight that "logical" connections between merchants are nuanced and often exceed simple industry categorization. However, the study also identifies practical hurdles:

  • Scalability: Manual labeling of tens of thousands of pairs is resource-intensive and difficult to scale as merchant catalogs grow. Future work should explore training predictive models on existing human labels to automate the validation process.
  • Unlabeled Data: There is potential to use matrix factorization on the human-response matrix to infer labels for unseen pairs.
  • Causal Complexity: Moving beyond simple item-to-item pairs toward more complex causal explanations remains a key area for future research.

In conclusion, this framework provides a vital tool for measuring the human-centric effectiveness of recommendation explanations, moving the field toward more trustworthy and logical AI systems.

Summary updated

BibTeX
@inproceedings{krasilnikov2023utilising,
  author = {Denis Krasilnikov and Oleg Lashinin and Maksim Tsygankov and Marina Ananyeva and Sergey Kolesnikov},
  title = {{Utilising Crowdsourcing to Assess the Effectiveness of Item-based Explanations of Merchant Recommendations}},
  booktitle = {CSW@WSDM},
  year = {2023},
  url = {https://openreview.net/forum?id=mVRNGHASO4},
}
Figure