Estimating Mobile-Friendliness Scores of Web Pages

Jihwan Song, Xing Xie, Yoon-Joon Lee, Ji-Rong Wen

This paper, published in 2010, presents a systematic approach to quantifying "mobile-friendliness"—a concept we take for granted today but was a significant hurdle in the era of early smartphones and PDAs. The researchers developed MFScorer, a machine-learning-based tool that evaluates webpages across 21 specific factors to predict how usable they will be on a mobile device.

TL;DR

The researchers developed MFScorer, a Support Vector Machine (SVM) model that automatically scores the mobile-friendliness of webpages based on 21 derived design factors. By integrating these scores into search engine results, they significantly improved mobile user satisfaction compared to existing tools like mobiReady.

What Problem Does This Paper Address?

In 2010, the mobile web was a frustrating place. Most websites were designed for large desktop monitors, high-speed cables, and powerful CPUs. When accessed via the "small display areas, slow communication links, and low computing power" of mobile devices, these pages often broke.

General-purpose search engines at the time didn't explicitly account for mobile-friendliness in their rankings. While some "mobile-only" search engines existed, they focused strictly on mobile-specific formats (like WML or cHTML), ignoring the vast majority of the web. The authors identified a need for a quantitative metric that could tell a search engine: "This standard HTML page will actually work well on a phone."

Methods

The authors followed a three-step process:

  1. Usability Study: They tasked 16 participants with examining 400 webpages to identify specific "obstacles." They categorized these into issues like broken layouts, unsupported tags (e.g., Flash, iframes), and the need for horizontal scrolling.
  2. Feature Engineering: From the study, they derived 21 "mobile-friendliness factors" (F1–F21). These included physical dimensions (page width/height), technical constraints (file size, unsupported tags), and readability metrics.
  3. The Scorer (MFScorer): They trained an SVM-based classifier to categorize pages into four levels: Poor (1), Bad (2), Good (3), and Excellent (4).
The MFScorer interface displaying the 21 extracted features and the resulting mobile-friendliness score for a sample webpage.

Key Insights and Ideas

  • The "Image Text" Problem: One of the most interesting factors identified was the use of text within images. While common in desktop design for aesthetic reasons, these images become illegible on small screens. The authors created a specific ratio (F19) to detect this.
  • Moving Beyond Binary: Unlike the W3C MobileOK Checker, which only provided a pass/fail based on best practices, MFScorer provides a quantitative score. This allows search engines to "nudge" rankings rather than just filtering pages out.
  • Human-Model Alignment: The core "aha moment" was realizing that existing automated tools (like mobiReady) had a significant gap between their scores and actual human satisfaction. By training on human-scored data, MFScorer achieved much higher alignment with real-world usability.

Key Results

  • Superior Accuracy: MFScorer’s automated scores were significantly more similar to human-generated scores than those produced by mobiReady.
  • Improved Search Relevance: Using a re-ranking formula that blends search engine rank with the MFScorer result (weighted by factor $\alpha$), MFScorer consistently outperformed the baseline as the weight of the mobile-friendliness score increased.
Performance comparison showing MFScorer's superior precision over mobiReady as the mobile-friendliness weight (α) increases.

Limitations and Future Work

While groundbreaking for its time, the paper focuses on era-specific constraints like slow communication links and obsolete technologies like Flash. However, the methodology of using human-centric usability studies to train automated scoring models remains a foundational principle in modern web design and SEO. The authors suggested that these scores could be integrated into live search engines to help users find information more efficiently on the go.

Summary updated

BibTeX
@article{song2010estimating,
  author = {Jihwan Song and Xing Xie and Yoon-Joon Lee and Ji-Rong Wen},
  title = {{Estimating Mobile-Friendliness Scores of Web Pages}},
  journal = {IEICE Trans. Information and Systems},
  year = {2010},
  doi = {10.1587/transinf.e93.d.3124},
}
Figure