A Platform for Streaming Push Notifications to Mobile Assessors
Lists
Only you see this
Abstract
We present an assessment platform for gathering online relevance judgments for mobile push notifications that will be deployed in the newly-created TREC 2016 Real-Time Summarization (RTS) track. There is emerging interest in building systems that filter social media streams such as tweets to identify interesting and novel content in real time, putatively for delivery to users' mobile phones. In our evaluation design, all participants subscribe to the Twitter streaming API to identify relevant tweets with respect to a set of interest profiles. As the systems generate results, they are pushed in real time to our evaluation broker via a REST API. The broker then "routes" the tweets to assessors who have installed a custom app on their mobile phones. We detail the design of this platform and discuss a number of challenges that need to be tackled in this type of "Living Labs" setup. It is our goal that such an evaluation design will mitigate any issues that have arisen in traditional batch-style evaluations of this type of task.
BibTeX
@inproceedings{roegiest2016platform,
author = {Adam Roegiest and Luchen Tan and Jimmy J. Lin and Charles L. A. Clarke},
title = {{A Platform for Streaming Push Notifications to Mobile Assessors}},
booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
year = {2016},
doi = {10.1145/2911451.2911463},
}Loading related content...