Introduction
In the early 2000s, information retrieval research primarily focused on keyword matching and the relevance of formal, static documents like academic papers. However, the emergence of the World Wide Web introduced a diverse array of multimedia, informal text, and complex hyperlinked structures. This shift necessitated a deeper understanding of how human searchers evaluate the utility of web pages beyond simple textual relevance. The 2003 SIGIR paper "Searchers' Criteria For Assessing Web Pages" by Tombros, Ruthven, and Jose addresses this gap by identifying the multi-faceted cues—textual, structural, and qualitative—that influence a user's decision to stay on a page or abandon it.
Methodology
To investigate these criteria, the researchers conducted a user study involving 24 participants. The study utilized Simulated Work-Task Situations to provide participants with a personal stake in their searches, moving beyond abstract queries. Participants engaged in three distinct search types:
- Background Search: Gathering general information on a topic.
- Decision Task: Evaluating options to make a specific choice (e.g., purchasing a product).
- Many Items Search: Compiling a comprehensive list of relevant items.
Data was collected using a "think-aloud" protocol, where participants narrated their decision-making processes. These sessions were recorded via video capture software to track screen actions and speech. One researcher then manually analyzed the sessions to extract mentions of specific page features that positively or negatively impacted the perceived utility of 862 assessed web pages.
Categorization of Assessment Criteria
The analysis identified 24 distinct features that searchers use to judge web pages. These features were grouped into five overarching categories: Text, Structure, Quality, Non-textual, and Physical Properties.
This categorization highlights that while text remains a primary driver, structural elements like layout and links, as well as qualitative factors like authority and recency, play significant roles in the evaluation process.
Frequency of Criteria Mentions
The study recorded a total of 1,602 individual mentions of page features. The distribution of these mentions reveals a hierarchy of importance among the criteria.
As shown in the data, Text was the most prominent category, accounting for 44.7% of all mentions. Within this category, "Content" was the single most cited feature (~24%), followed by the presence of "Numbers" (~10%). Structure followed at 21.6%, with Quality at 17.7%. Non-textual elements (8.9%) and Physical Properties (7.1%) were less frequent but still notable factors.
Key Insights
The Importance of Quantitative Data
A significant finding was the high value users placed on "Numbers." Quantitative data often served as a primary indicator of a page's utility, suggesting that searchers look for concrete, verifiable information to support their tasks.
Positive vs. Negative Filters
The study distinguished between features that attract users and those that act as "deal-breakers." Certain criteria, such as Link Quality, General Quality, and Scope/Depth, were mentioned significantly more often when they were perceived as poor. These "negative signals" allow users to quickly filter out and discard irrelevant or low-quality pages.
The "Sole Indicator" Phenomenon
Interestingly, for approximately 43% of the pages assessed, users relied on a single feature to make a judgment. While "Content" was the most frequent solo indicator, other features like "Query Terms" and "Link Quality" also served as the sole basis for accepting or rejecting a page.
Conclusion
This research demonstrates that web page assessment is a complex process involving a variety of cues. While textual content is paramount, structural and qualitative features are essential components of the user's mental model of relevance. Understanding these 24 criteria can inform the design of more sophisticated search engines, recommendation systems, and automated summarization tools that better align with human evaluative behavior.
The authors note that this exploratory study, while insightful, is limited by its sample size and potential subjective bias in manual categorization. Future research is suggested to explore how these criteria evolve throughout different stages of a search task.
Summary updated