Online Evaluation of Point-Of-Interest Recommendation Systems

Adriel Dean-Hall, Charles L. A. Clarke, Jaap Kamps, Julia KiselevaSCST@ECIR 2015 · OpenReview

TL;DR

This paper introduces a novel online evaluation framework for point-of-interest (POI) recommendation systems, moving beyond traditional batch evaluations. By enabling live interactions between crowd-sourced assessors and multiple recommendation services, the system aims to provide more realistic and continuous feedback for system improvement.

What Problem Does This Paper Address?

Evaluating recommendation systems, particularly for POIs, has traditionally been done in a batch fashion. This involves participants submitting results for a predefined set of topics, which are then evaluated by assessors. While this method has been used in tracks like TREC Contextual Suggestion, it suffers from significant drawbacks: long waiting times for suggestions, broken assessor engagement over extended periods, and difficulty in retaining assessors. This delay makes it harder for assessors to recall previous interactions and for systems to receive timely feedback for iterative improvement. The paper addresses the need for a more dynamic and efficient evaluation methodology that mirrors real-world usage and facilitates continuous learning for recommendation services.

Methods

The core contribution is a live evaluation system where recommendation services are exposed online and respond to suggestion requests in real-time. Crowd-sourced assessors interact with merged results from multiple services. The system works as follows:

  1. Request Initiation: A user (or assessor) makes a request for POI suggestions, providing their profile and a city context.
  2. Service Querying: The evaluation system sends this request to a subset of registered recommendation services.
  3. Response Merging: Services respond with ranked lists of POI suggestions. These lists are merged into a single, unified list presented to the user.
  4. Live Interaction & Feedback: The user interacts with the merged suggestions (e.g., by viewing details, starring, or rating them). This interaction data is captured.
  5. Continuous Scoring: Based on user interactions, each participating service's performance is continuously updated and scored. This allows for dynamic ranking and feedback.

A key aspect is the Service Interface, which requires services to respond to requests with a list of attraction IDs. The system also defines a User Profile that expands over time based on user interactions, allowing for increasingly personalized recommendations. The paper also discusses a Dataset Collection approach where POIs are drawn from a fixed collection, simplifying evaluation and increasing reusability of judgments.

Datasets

The paper mentions that for this track, the points-of-interest (POIs) that services can recommend come from a fixed collection of attractions. This collection is being curated by research groups involved in previous TREC Contextual Suggestion tracks. Services will return attraction IDs, and when displayed to users, these will include a title, short description, and website URL. The authors note that additional information like category or reviews could be included in the future.

Key Insights and Ideas

  • Moving from Batch to Live Evaluation: The most significant insight is the shift from static, batch evaluations to a dynamic, live system. This better reflects how users interact with recommendation systems in the real world and provides immediate feedback loops.
  • Continuous Scoring and Feedback: The system allows for continuous scoring of services, enabling them to adapt and improve their strategies throughout the experiment, rather than just receiving a final score at the end.
  • The Importance of User Profile Evolution: The paper highlights that user profiles are not static but evolve with user interactions. This dynamic profile is crucial for recommendation services to learn and adapt their suggestions over time.
  • Challenges of Merging Results: A practical challenge identified is the need for effective methods to merge suggestions from multiple services into a coherent list for the user, drawing on existing work like team drafting.
  • The Role of Assessors: The paper acknowledges the reliance on crowd-sourced assessors for evaluation, similar to previous TREC tracks, and discusses how their interactions drive the scoring mechanism.

Key Results

The paper does not present quantitative results from a completed experiment. Instead, it outlines the design and methodology for an upcoming evaluation system. The key "results" are the proposed framework and its features:

  • A system designed for online evaluation of POI recommendation services.
  • A mechanism for live interaction between assessors and multiple services.
  • A continuous scoring system to track service performance dynamically.
  • A defined service interface and user profile structure.
  • Plans for using a fixed dataset of attractions for consistent evaluation.

Limitations and Future Work

  • Service Selection Strategy: The paper notes that initially, services are selected randomly for each request. However, they acknowledge that a more sophisticated strategy could be employed, such as prioritizing services with similar performance to better differentiate them, or de-prioritizing consistently poor performers. This is left for future work.
  • Personalized Descriptions: While services can optionally provide personalized descriptions for attractions, the evaluation of these descriptions is separate from the main service performance evaluation.
  • Scalability of Live Evaluation: The paper implicitly suggests that managing a live system with many services and assessors presents scalability challenges, though not explicitly detailed as a limitation.
  • Future Work: The authors suggest that the system could be extended to provide scores or leaderboards more frequently to services. They also mention the potential for real users to use the system outside of the track experiments, allowing for continuous feedback. Furthermore, they envision exploring different methods for presenting results to users, including mobile applications, to cater to mobile users.

Summary updated

BibTeX
@inproceedings{deanhall2015online,
  author = {Adriel Dean-Hall and Charles L. A. Clarke and Jaap Kamps and Julia Kiseleva},
  title = {{Online Evaluation of Point-Of-Interest Recommendation Systems}},
  booktitle = {SCST@ECIR},
  year = {2015},
  url = {https://openreview.net/forum?id=6YjmlIa-B7Q},
}
Figure