Detecting success in mobile search from interaction

Qi Guo, Qi Guo 0002, Shuai Yuan, Eugene AgichteinSIGIR 2011 · DOI

TL;DR

This paper tackles the challenge of automatically predicting whether a user has successfully completed a mobile search task and is satisfied with the experience. By analyzing fine-grained client-side interaction signals (like page scrolling and zooming) in addition to traditional server-side metrics, the authors achieve nearly 80% accuracy in predicting search success, significantly outperforming previous models. This work is crucial for improving mobile search engine performance and user experience.

What Problem Does This Paper Address?

Evaluating the effectiveness of search engines is vital for their improvement and for providing real-time assistance to users. While this has been studied extensively for desktop search, the rapid rise of mobile devices presents a unique challenge. Mobile search behavior differs significantly from desktop search due to factors like smaller screens, touch interfaces, and different usage contexts. Existing models, often relying on desktop interaction signals, may not translate well to the mobile environment. This paper aims to bridge this gap by investigating how to predict mobile search success and satisfaction using interaction data captured directly from mobile devices.

Methods

The authors conducted a user study where participants performed various mobile search tasks using a modified Chrome browser on Android phones. This browser logged client-side interaction events, such as the number of pages browsed, zooming, scrolling/sliding, and device orientation changes. These client-side features were combined with traditional server-side features like the number of queries, clicks, and task duration.

The prediction task was framed as a binary classification problem: classifying each search task as either successful/satisfied or unsuccessful/unsatisfied. They experimented with several classification algorithms, ultimately reporting results for the Bayes Network (BN) classifier, which performed best. Ten-fold cross-validation was used to evaluate the models.

Datasets

The paper describes a user study conducted with 10 participants (undergraduate and graduate students/staff at Emory University) who performed 8 search tasks each. The tasks were designed to be representative of common mobile search needs, including geographical intents. The study collected both task outcomes (success/satisfaction ratings) and detailed client-side interaction logs from a modified Chrome browser. This constitutes a newly introduced benchmark dataset for mobile search interaction analysis.

Key Insights and Ideas

  • Client-side signals are crucial for mobile search: The study highlights that fine-grained interactions like scrolling, zooming, and page browsing on a touch screen provide significant predictive power for search success, often more so than traditional server-side metrics alone.
  • Effort correlates with dissatisfaction: Generally, more effort expended by the user (indicated by more browsed pages, longer task duration, etc.) correlates with lower satisfaction and success, which is intuitive given the constraints of mobile devices.
  • Subjectivity in satisfaction: While success and satisfaction are correlated, they are not perfectly so. The variance in satisfaction ratings across users for the same task suggests a higher degree of subjectivity in judging satisfaction compared to success.
  • Task difficulty influences variance: Easier tasks tend to have less variance in success and satisfaction ratings across users, while more difficult tasks show greater variation, indicating opportunities for personalized assistance.

Key Results

The Bayes Network classifier, incorporating both client-side and server-side features, achieved nearly 80% accuracy in predicting search success. Specifically, the "Full" model (combining both feature sets) achieved 78.0% accuracy for predicting success and 78.8% for predicting satisfaction. Notably, client-side features alone outperformed server-side features for predicting success (80.0% accuracy vs. 66.3%), while the combination of features yielded the best performance for predicting satisfaction. This significantly outperforms a baseline model that achieved around 54-58% accuracy.

Limitations and Future Work

The authors acknowledge that the study was conducted with a relatively small sample size (10 participants). They suggest that future work could involve larger-scale studies with a more diverse user population and a wider range of mobile search tasks. Further exploration into the specific contributions of individual client-side features and their interplay could also be beneficial. The paper lays the groundwork for more extensive research into understanding and improving the mobile search experience.

Summary updated

BibTeX
@inproceedings{guo2011detecting,
  author = {Qi Guo and Qi Guo and Shuai Yuan and Eugene Agichtein},
  title = {{Detecting success in mobile search from interaction}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2011},
  doi = {10.1145/2009916.2010133},
}

Loading related content...

Figure