Abstract: We present a system that offers a new way of assessing web document relevance and a new approach to the web-based evaluation of such a system. Provisionally named WebDocSum, the system is a query-biased web page summariser that aims to provide an alternative to the short, irrelevant abstracts typical of many web search result lists. Based on an initial evaluation, the system appears to be more useful in helping users gauge document relevance than the traditional ranked titles/abstracts approach.
Introduction: The "Searcher's Slog"
In the landscape of early web search, users frequently encountered a phenomenon known as the "searcher's slog." Search engines typically provided static snippets—generic abstracts or the first few lines of a page—which often failed to indicate how a document actually related to a user's specific query. This forced users into a repetitive cycle of clicking results, finding them irrelevant, and returning to the results page. WebDocSum was developed to address this by providing "query-biased" summaries that highlight the specific context of search terms within a document.
The WebDocSum System
WebDocSum acts as an interface enhancement for major search engines (originally tested with Google and AltaVista). Unlike static descriptions, it generates dynamic summaries on the fly using a sentence extraction model. The algorithm prioritizes sentences based on:
- Position and Formatting: Sentences at the start of a document or those using bold text and titles are weighted more heavily.
- Query Density: The frequency of search terms within a sentence.
- Query Context: A specialized scoring system that ensures the summary reflects the user's specific information need.
In the user interface, these summaries appear in a "Summary Window" when a user hovers over a search result. This window also provides metadata, such as the number of links, the first non-text object (e.g., an image), and the total document size, to help users build a mental model of the page before visiting it.
Methodology
The system was evaluated using 24 participants ranging from novices to experts. Rather than relying on automated metrics, the authors employed "Simulated Work Tasks" to measure real-world utility. These tasks included fact-finding (e.g., locating an email address), item-finding (e.g., finding hotels in Paris), decision-based searching, and background research.
Four configurations were tested:
- System A: Standard Google
- System B: Standard AltaVista
- System C: Google enhanced with WebDocSum
- System D: AltaVista enhanced with WebDocSum
Key Results
The subjective evaluation utilized a 5-point scale where 1 represented the best possible score and 5 the worst. The results indicated a clear preference for query-biased summarization.
As shown in the data, the enhanced systems (C and D) consistently outperformed the standard baselines (A and B) across all metrics. For instance, System D achieved a relevance score of 1.67, a significant improvement over System B’s 3.54. Users also found the query-biased summaries to be far more "complete," with scores ranging from 1.91 to 2.21, compared to the 3.63 to 3.75 range for standard snippets. Ultimately, 23 out of 24 participants selected a WebDocSum-enhanced system as their preferred search interface.
Conclusion and Future Work
While the prototype faced latency issues—taking approximately 7 seconds to generate summaries—the study proved that query-biased context is vital for efficient information retrieval. By reducing cognitive load and providing a stronger "information scent," WebDocSum allows users to make more accurate relevance judgments without unnecessary clicks. Future iterations aim to integrate these summaries directly into search engine indices to improve speed and further explore the role of non-textual elements in relevance assessment.
Summary updated