Searchers' criteria For assessing web pages

Anastasios Tombros, Ian Ruthven, Joemon M. Jose

Introduction

In the early 2000s, information retrieval research primarily focused on keyword matching and the relevance of formal, static documents like academic papers. However, the emergence of the World Wide Web introduced a diverse array of multimedia, informal text, and complex hyperlinked structures. This shift necessitated a deeper understanding of how human searchers evaluate the utility of web pages beyond simple textual relevance. The 2003 SIGIR paper "Searchers' Criteria For Assessing Web Pages" by Tombros, Ruthven, and Jose addresses this gap by identifying the multi-faceted cues—textual, structural, and qualitative—that influence a user's decision to stay on a page or abandon it.

Methodology

To investigate these criteria, the researchers conducted a user study involving 24 participants. The study utilized Simulated Work-Task Situations to provide participants with a personal stake in their searches, moving beyond abstract queries. Participants engaged in three distinct search types:

  1. Background Search: Gathering general information on a topic.
  2. Decision Task: Evaluating options to make a specific choice (e.g., purchasing a product).
  3. Many Items Search: Compiling a comprehensive list of relevant items.

Data was collected using a "think-aloud" protocol, where participants narrated their decision-making processes. These sessions were recorded via video capture software to track screen actions and speech. One researcher then manually analyzed the sessions to extract mentions of specific page features that positively or negatively impacted the perceived utility of 862 assessed web pages.

Categorization of Assessment Criteria

The analysis identified 24 distinct features that searchers use to judge web pages. These features were grouped into five overarching categories: Text, Structure, Quality, Non-textual, and Physical Properties.

Classification of the 24 criteria used by searchers into five main categories.

This categorization highlights that while text remains a primary driver, structural elements like layout and links, as well as qualitative factors like authority and recency, play significant roles in the evaluation process.

Frequency of Criteria Mentions

The study recorded a total of 1,602 individual mentions of page features. The distribution of these mentions reveals a hierarchy of importance among the criteria.

Distribution of mentions across the identified criteria, highlighting the dominance of content and numbers.

As shown in the data, Text was the most prominent category, accounting for 44.7% of all mentions. Within this category, "Content" was the single most cited feature (~24%), followed by the presence of "Numbers" (~10%). Structure followed at 21.6%, with Quality at 17.7%. Non-textual elements (8.9%) and Physical Properties (7.1%) were less frequent but still notable factors.

Key Insights

The Importance of Quantitative Data

A significant finding was the high value users placed on "Numbers." Quantitative data often served as a primary indicator of a page's utility, suggesting that searchers look for concrete, verifiable information to support their tasks.

Positive vs. Negative Filters

The study distinguished between features that attract users and those that act as "deal-breakers." Certain criteria, such as Link Quality, General Quality, and Scope/Depth, were mentioned significantly more often when they were perceived as poor. These "negative signals" allow users to quickly filter out and discard irrelevant or low-quality pages.

The "Sole Indicator" Phenomenon

Interestingly, for approximately 43% of the pages assessed, users relied on a single feature to make a judgment. While "Content" was the most frequent solo indicator, other features like "Query Terms" and "Link Quality" also served as the sole basis for accepting or rejecting a page.

Conclusion

This research demonstrates that web page assessment is a complex process involving a variety of cues. While textual content is paramount, structural and qualitative features are essential components of the user's mental model of relevance. Understanding these 24 criteria can inform the design of more sophisticated search engines, recommendation systems, and automated summarization tools that better align with human evaluative behavior.

The authors note that this exploratory study, while insightful, is limited by its sample size and potential subjective bias in manual categorization. Future research is suggested to explore how these criteria evolve throughout different stages of a search task.

Summary updated

Abstract

We investigate the criteria used by online searchers when assessing the relevance of web pages to information-seeking tasks. Twenty four searchers were given three tasks each, and indicated the features of web pages which they employed when deciding about the usefulness of the pages. These tasks were presented within the context of a simulated work-task situation. The results of this study provide a set of criteria used by searchers to decide about the utility of web pages. Such criteria have implications for the design of systems that use or recommend web pages, as well as to authors of web pages.
BibTeX
@inproceedings{tombros2003searchers,
  author = {Anastasios Tombros and Ian Ruthven and Joemon M. Jose},
  title = {{Searchers' criteria For assessing web pages}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2003},
  doi = {10.1145/860435.860513},
}
Figure