Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems
Lists
Only you see this
Abstract
Real-time push notification systems monitor continuous document streams such as social media posts and alert users to relevant content directly on their mobile devices. We describe a user study of such systems in the context of the TREC 2016 Real-Time Summarization Track, where system updates are immediately delivered as push notifications to the mobile devices of a cohort of users. Our study represents, to our knowledge, the first deployment of an interleaved evaluation framework for prospective information needs, and also provides an opportunity to examine user behavior in a realistic setting. Results of our online in-situ evaluation are correlated against the results a more traditional post-hoc batch evaluation. We observe substantial correlations between many online and batch evaluation metrics, especially for those that share the same basic design (e.g., are utility-based). For some metrics, we observe little correlation, but are able to identify the volume of messages that a system pushes as one major source of differences.
BibTeX
@inproceedings{roegiest2017online,
author = {Adam Roegiest and Luchen Tan and Jimmy Lin},
title = {{Online In-Situ Interleaved Evaluation of Real-Time Push Notification Systems}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2017},
doi = {10.1145/3077136.3080808},
}Loading related content...