Keynote

Nan Guan, Yiqun LiuEvia@ntcir 2016

Introduction

For decades, the Information Retrieval (IR) community has relied on relevance-based metrics such as Mean Average Precision (MAP) and Normalized Discounted Cumulative Gain (nDCG) to evaluate the performance of search engines. These metrics operate on a fundamental assumption: if a document matches a query, it is "good." However, as search technology has evolved, a significant gap has emerged between technical relevance and actual user satisfaction. This paper explores the complex relationship between relevance and usefulness, the impact of modern heterogeneous search results, and a novel framework for predicting user happiness through behavioral signals.

The Three Pillars: Relevance, Usefulness, and Satisfaction

A core challenge in modern search evaluation is the divergence between professional relevance annotations and user-perceived usefulness. While a professional assessor might mark a document as highly relevant because it contains the correct keywords or topics, that same document may fail to help a specific user complete their task.

Research into this area identifies three distinct pillars of evaluation:

  1. Relevance: As determined by professional assessors based on topical matching.
  2. Usefulness: As perceived by the actual user in the context of their specific information need.
  3. Satisfaction: The overall emotional and cognitive success of the search session.

Understanding the correlation—or lack thereof—between these pillars is essential for building search engines that do more than just find documents; they must solve problems.

The Impact of Heterogeneous Search Environments

Modern Search Engine Result Pages (SERPs) have moved far beyond the traditional list of "ten blue links." Today’s environments are heterogeneous, incorporating various "vertical" results such as image galleries, news boxes, maps, and knowledge graphs.

This environmental shift fundamentally alters how users evaluate the quality of a search page. The presence of vertical results changes the visual flow and the cognitive load of the user. Research suggests that these diverse elements do not just provide additional information; they change the user's internal threshold for what constitutes a "successful" search. Evaluating these complex layouts requires a move away from linear ranking metrics toward holistic page-level assessments.

Decoding User Behavior: Mouse Movement Motifs

To bridge the gap between system logs and user satisfaction, researchers are looking beyond simple metrics like click-through rates or dwell time. While clicks indicate interest, they do not necessarily indicate success—a user might click a result and immediately realize it is irrelevant (a "bounce").

A novel approach to this problem involves analyzing the geometry of mouse movements, referred to as "motifs." By identifying specific behavioral signatures—patterns in how a user moves their cursor across the screen—researchers can identify non-intrusive signals of frustration or satisfaction. These mouse motifs serve as a rich proxy for user intent and experience, allowing for a more nuanced prediction of whether a user is actually happy with their search session without requiring intrusive surveys.

Conclusion

As search engines become more integrated into complex task-solving, the metrics used to evaluate them must also evolve. By recognizing that relevance is not synonymous with satisfaction and by leveraging sophisticated behavioral signals like mouse motifs, the IR community can move closer to a truly user-centric model of search evaluation. Future work continues to refine these behavioral models to better understand the user experience in increasingly diverse and interactive search environments.

Summary updated

BibTeX
@inproceedings{guan2016keynote,
  author = {Nan Guan and Yiqun Liu},
  title = {{Keynote}},
  booktitle = {Evia@ntcir},
  year = {2016},
}
Figure