Finding relevant documents using top ranking sentences: an evaluation of two alternative schemes

Ryen White, Ian Ruthven, Joemon M. Jose

Introduction

Web search behavior is often characterized by a "minimalist" approach. Users typically submit short queries and rarely explore beyond the first page of results. If the initial top 10 results do not satisfy the information need, users are more likely to reformulate their query than to scroll down. This behavior is cognitively taxing, as generating effective new search terms is difficult, and valuable information often remains hidden just below the fold. This paper investigates whether presenting users with a list of the most relevant sentences extracted from the top 30 search results can provide a stronger "information scent," encouraging deeper exploration of the retrieved set.

Methodology and System Design

The researchers developed three experimental systems to test different levels of information delivery:

  1. System 1 (Baseline): Provides a four-sentence summary of a document when a user hovers over its title in the search results.
  2. System 2 (Top-Ranking Sentences): Displays a separate, static list of high-value sentences extracted from the entire top 30 documents, regardless of their rank.
  3. System 3 (Implicit Feedback): Uses reading time as an implicit signal of interest. By monitoring how long a user dwells on a summary, the system identifies relevant content and uses Robertson’s $wpq$ formula to dynamically re-rank the "top-ranking sentences" list in real-time.
The experimental search interface showing the standard result list (left) and the extracted top-ranking sentences (right).

The study involved 24 participants (split between experienced and inexperienced searchers) performing three types of tasks: fact-finding, background research, and decision-making. The systems were implemented as wrappers around the AltaVista search engine.

Key Insights: Reading Time and User Interaction

A central premise of the implicit feedback system (System 3) is that reading behavior reflects document relevance. The study confirmed this, finding that users spend approximately 35% more time per character reading summaries they later mark as relevant compared to those they find irrelevant.

Comparison of average reading time per character for relevant versus non-relevant document summaries.

This validation of "dwell time" suggests that sentence-level interaction can serve as a powerful proxy for user interest without requiring explicit ratings. Furthermore, providing full sentences offers "local context," allowing users to understand the relationship between their query terms and the document content before committing to a click.

Results and Discussion

The introduction of top-ranking sentences significantly altered search behavior:

  • Increased Exploration: Users of Systems 2 and 3 viewed nearly twice as many documents ranked 11th or lower (8.1 documents on average) compared to the baseline (4.6 documents).
  • Improved Efficiency: Participants submitted 30% fewer queries, spending more time extracting value from the initial results rather than struggling with query reformulation.
  • System Preference: System 2 was the most preferred. While System 3 helped inexperienced searchers bridge the skill gap by refining their queries automatically, the dynamic re-ranking was often jarring. Experienced users found the "disappearing content"—where sentences they were reading were suddenly replaced by new ones—to be a significant hindrance.

Conclusion

Presenting top-ranking sentences from a broad set of search results effectively breaks the "first-page trap." While implicit feedback based on reading time shows promise for assisting novice users, the stability of the user interface remains a critical factor for user satisfaction. Future work should focus on less intrusive methods of updating dynamic content to maintain the user's focus while providing real-time relevance adjustments.

Summary updated

Abstract

In this paper we present an evaluation of techniques that are designed to encourage web searchers to interact more with the results of a web search. Two specific techniques are examined: the presentation of sentences that highly match the searcher's query and the use of implicit evidence. Implicit evidence is evidence captured from the searcher's interaction with the retrieval results and is used to automatically update the display. Our evaluation concentrates on the effectiveness and subject perception of these techniques. The results show, with statistical significance, that the techniques are effective and efficient for information seeking.
BibTeX
@inproceedings{white2002finding,
  author = {Ryen White and Ian Ruthven and Joemon M. Jose},
  title = {{Finding relevant documents using top ranking sentences: an evaluation of two alternative schemes}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2002},
  doi = {10.1145/564376.564389},
}
Figure