Random Surfers on a Web Encyclopedia

Florian Geigl, Daniel Lamprecht, Rainer Hofmann-Wellenhof, Simon Walk, Markus Strohmaier, Denis Helic

Abstract

The random surfer model is a frequently used model for simulating user navigation behavior on the Web. Various algorithms, such as PageRank, are based on the assumption that the model represents a good approximation of users browsing a website. However, the way users browse the Web has been drastically altered over the last decade due to the rise of search engines. Hence, new adaptations for the established random surfer model might be required, which better capture and simulate this change in navigation behavior. In this article we compare the classical uniform random surfer to empirical navigation and page access data in a Web Encyclopedia. Our high level contributions are (i) a comparison of stationary distributions of different types of the random surfer to quantify the similarities and differences between those models as well as (ii) new insights into the impact of search engines on traditional user navigation. Our results suggest that the behavior of the random surfer is almost similar to those of users - as long as users do not use search engines. We also find that classical website navigation structures, such as navigation hierarchies or breadcrumbs, only exercise limited influence on user navigation anymore. Rather, a new kind of navigational tools (e.g., recommendation systems) might be needed to better reflect the changes in browsing behavior of existing users.

Introduction

The "random surfer" model has long served as the mathematical foundation for understanding web navigation and ranking page importance. The core assumption—that users traverse the web by clicking links at random and occasionally "teleporting" to new pages—powered the early success of algorithms like Google’s PageRank. However, the modern web landscape is dominated by search engines. Instead of starting at a homepage and drilling down through a site's hierarchy, users frequently land directly on deep content pages via external search queries. This study investigates whether the classic random surfer model still accurately reflects human behavior in this search-centric era.

Methodology and Data

To evaluate the model, researchers analyzed 59 days of server logs from Austria-Forum, a large web encyclopedia. The data was meticulously preprocessed to remove bot traffic and administrative actions, focusing on human-readable content.

One critical step in the methodology was "sessionization"—defining the boundaries of a single user visit. By analyzing how different time thresholds affected the number of sessions and average clicks per session, the researchers identified a 30-minute inactivity window as the optimal "Time Delta" for their analysis.

Sessionization analysis and session length distribution

The study compared three distinct models using their stationary distributions (the long-term probability of a surfer being on any given page):

  • Uniform Random Surfer: The classic model following links with equal probability.
  • Pragmatic Random Surfer: A model weighted by actual human clickstream data.
  • Lateral Random Surfer: A distribution representing actual page views, including direct entries from external search engines.

Key Results

Modeling Internal Navigation

The research found that for users already navigating within the site, the classic random surfer is a remarkably resilient proxy for human behavior. The Uniform and Pragmatic models showed a near-perfect Pearson correlation of 0.98. This suggests that the internal link structure of a website is a powerful predictor of how humans will move between pages once they have arrived.

The Search Engine Gap

While the random surfer excels at modeling internal link-following, it fails to capture how users arrive at the site. The stationary distributions of the internal models (Uniform and Pragmatic) showed weak correlations ($\rho = 0.38$ and $0.47$, respectively) with the Lateral model.

Comparison of stationary distributions across the three surfer models

As visualized in the scatter plots above, the Uniform and Pragmatic models align closely (a), but both diverge significantly from the Lateral model (b and c). Search engines act as "teleportation" mechanisms that bypass traditional site hierarchies, distributing users across the "long tail" of specific, deep-nested articles rather than concentrating them in high-traffic hubs.

Distribution Inequality

The study used Gini coefficients to measure the inequality of page visits. The Uniform surfer resulted in a very high Gini coefficient of 0.96, indicating that traffic was extremely concentrated on a few central pages. In contrast, the Lateral surfer (representing real-world access) had a much lower Gini of 0.70, proving that search engines effectively equalize traffic by driving users to a more diverse set of pages across the entire site.

Conclusion and Implications

The findings suggest that while the random surfer model remains a valid tool for understanding internal link navigation, it must be adapted to account for the massive influence of search engines. Traditional navigational aids like breadcrumbs and rigid hierarchies are becoming less influential as users increasingly "teleport" to leaf nodes.

For website administrators, this shift suggests a move away from focusing solely on hierarchical navigation. Instead, there is a growing need for robust recommendation systems and related-content links to engage users who land deep within a site and might otherwise leave immediately after finding their specific answer. Future research will look to validate these findings on even larger datasets, such as the English Wikipedia, to see if these patterns hold across different scales of web encyclopedias.

Summary updated

Abstract

The random surfer model is a frequently used model for simulating user navigation behavior on the Web. Various algorithms, such as PageRank, are based on the assumption that the model represents a good approximation of users browsing a website. However, the way users browse the Web has been drastically altered over the last decade due to the rise of search engines. Hence, new adaptations for the established random surfer model might be required, which better capture and simulate this change in navigation behavior. In this article we compare the classical uniform random surfer to empirical navigation and page access data in a Web Encyclopedia. Our high level contributions are (i) a comparison of stationary distributions of different types of the random surfer to quantify the similarities and differences between those models as well as (ii) new insights into the impact of search engines on traditional user navigation. Our results suggest that the behavior of the random surfer is almost similar to those of users - as long as users do not use search engines. We also find that classical website navigation structures, such as navigation hierarchies or breadcrumbs, only exercise limited influence on user navigation anymore. Rather, a new kind of navigational tools (e.g., recommendation systems) might be needed to better reflect the changes in browsing behavior of existing users.
BibTeX
@inproceedings{geigl2015random,
  author = {Florian Geigl and Daniel Lamprecht and Rainer Hofmann-Wellenhof and Simon Walk and Markus Strohmaier and Denis Helic},
  title = {{Random Surfers on a Web Encyclopedia}},
  booktitle = {I-KNOW},
  year = {2015},
  url = {https://openreview.net/forum?id=_3laxQuXasi},
}
Figure