Introduction
Web search behavior is often characterized by a "minimalist" approach. Users typically submit short queries and rarely explore beyond the first page of results. If the initial top 10 results do not satisfy the information need, users are more likely to reformulate their query than to scroll down. This behavior is cognitively taxing, as generating effective new search terms is difficult, and valuable information often remains hidden just below the fold. This paper investigates whether presenting users with a list of the most relevant sentences extracted from the top 30 search results can provide a stronger "information scent," encouraging deeper exploration of the retrieved set.
Methodology and System Design
The researchers developed three experimental systems to test different levels of information delivery:
- System 1 (Baseline): Provides a four-sentence summary of a document when a user hovers over its title in the search results.
- System 2 (Top-Ranking Sentences): Displays a separate, static list of high-value sentences extracted from the entire top 30 documents, regardless of their rank.
- System 3 (Implicit Feedback): Uses reading time as an implicit signal of interest. By monitoring how long a user dwells on a summary, the system identifies relevant content and uses Robertson’s $wpq$ formula to dynamically re-rank the "top-ranking sentences" list in real-time.
The study involved 24 participants (split between experienced and inexperienced searchers) performing three types of tasks: fact-finding, background research, and decision-making. The systems were implemented as wrappers around the AltaVista search engine.
Key Insights: Reading Time and User Interaction
A central premise of the implicit feedback system (System 3) is that reading behavior reflects document relevance. The study confirmed this, finding that users spend approximately 35% more time per character reading summaries they later mark as relevant compared to those they find irrelevant.
This validation of "dwell time" suggests that sentence-level interaction can serve as a powerful proxy for user interest without requiring explicit ratings. Furthermore, providing full sentences offers "local context," allowing users to understand the relationship between their query terms and the document content before committing to a click.
Results and Discussion
The introduction of top-ranking sentences significantly altered search behavior:
- Increased Exploration: Users of Systems 2 and 3 viewed nearly twice as many documents ranked 11th or lower (8.1 documents on average) compared to the baseline (4.6 documents).
- Improved Efficiency: Participants submitted 30% fewer queries, spending more time extracting value from the initial results rather than struggling with query reformulation.
- System Preference: System 2 was the most preferred. While System 3 helped inexperienced searchers bridge the skill gap by refining their queries automatically, the dynamic re-ranking was often jarring. Experienced users found the "disappearing content"—where sentences they were reading were suddenly replaced by new ones—to be a significant hindrance.
Conclusion
Presenting top-ranking sentences from a broad set of search results effectively breaks the "first-page trap." While implicit feedback based on reading time shows promise for assisting novice users, the stability of the user interface remains a critical factor for user satisfaction. Future work should focus on less intrusive methods of updating dynamic content to maintain the user's focus while providing real-time relevance adjustments.
Summary updated