A Quantitative Approach in Heuristic Evaluation of E-commerce Websites

Xiaosong Li, Ye Liu, Zizhou Fan, Will LiarXiv 2018 · arXiv

Abstract

Traditional heuristic evaluation of user interfaces often suffers from subjective bias and poor scalability. This paper presents an automated, quantitative approach to evaluating e-commerce websites by mapping the "8C" design framework to measurable web elements. Through iterative training against real-world conversion rates, the authors developed a scoring model that objectively estimates UI/UX quality at scale.

Introduction

Heuristic evaluation is a cornerstone of user experience (UX) design, yet it remains fundamentally tied to human judgment. Frameworks like the 8C model—comprising Context, Content, Community, Customization, Communication, Connection, Commerce, and Collaboration—offer structured ways to assess e-commerce sites. However, manual reviews are slow, expensive, and inconsistent across different evaluators.

This research addresses these limitations by asking whether heuristic evaluation can be automated. By quantitatively measuring website elements such as HTML tags and keywords, the authors aim to objectively predict a site's effectiveness, using Conversion Rate (CR) as the primary metric for success.

Methodology: Quantifying the 8C Framework

The researchers developed a PHP-based web application designed to scrape and analyze website homepages. The core of the methodology lies in bridging qualitative UX dimensions with quantitative code analysis.

Mapping Dimensions to Code

Each of the eight dimensions in the 8C framework was mapped to specific, quantifiable web elements. For instance, the "Collaboration" dimension was identified through keywords like "FAQ," "Forum," or "Feedback." The "Context" dimension might be represented by structural HTML tags that define layout and organization.

The 8C framework dimensions and their operational meanings within the study.

Scoring and Iterative Training

A scoring mechanism was established where each identified tag or keyword was assigned a numeric weight. The score for a specific dimension was calculated by multiplying the frequency of these elements by their weights, then normalizing the result against the total number of HTML tags on the page to ensure fairness across sites of different sizes.

To refine these weights, the authors conducted ten iterative experiments. In each round, they scored a set of training websites, ranked them based on these scores, and compared the predicted ranking to the sites' actual Conversion Rates. Manual adjustments were made to weights and mappings where deviations occurred. Mathematical scaling factors were also introduced to prevent high-frequency dimensions, such as "Context," from disproportionately influencing the final aggregate score.

Experimental Setup

The study utilized three distinct datasets to train and validate the model:

  • Training Data: 100 e-commerce websites across five retail categories (Electronics, Publishing, Home & Garden, Books, Industrial Equipment).
  • Testing Data: Seven top-tier e-commerce websites from 2010 with known CRs used for iterative validation during the training phase.
  • Verification Data: Fifteen top e-commerce websites from 2014 (e.g., DollarShaveClub) used to test the final model's generalizability on unseen data.

Results and Discussion

The iterative training process demonstrated a clear trend of improvement. As weights were adjusted over the ten experiments, the model's predictions increasingly aligned with real-world performance metrics.

Average absolute difference between expected and actual outcomes across ten training experiments.

Key Findings

  • Convergence: The average absolute difference between the predicted ranking and the actual CR ranking steadily decreased, validating the iterative adjustment approach.
  • Generalization: When applied to the 2014 verification dataset, the model maintained a low error rate, suggesting that the identified code-based proxies for UX are robust over time.
  • Balanced Evaluation: By Experiment 10, scaling successfully balanced the dimensions. The contribution of "Context" dropped from an initial 40.94% to a more representative 14.71%, allowing other critical factors like "Content" and "Communication" to influence the final score.
  • Industry Context: The research highlighted that baseline CRs vary significantly by industry (e.g., ~23% for Electronics vs. ~7% for Industrial Equipment). This suggests that automated scores should be interpreted relative to industry-specific benchmarks.

Conclusion and Future Work

This study demonstrates that the presence and frequency of specific HTML tags and keywords can serve as a viable automated proxy for subjective UX evaluation. By grounding abstract design concepts in conversion data, the authors provide a standardized method for large-scale UI/UX assessment.

However, limitations remain. The current tool only analyzes homepages, missing critical journey steps like checkout flows. Future work aims to automate the weight-tuning process using machine learning and to expand the tool's output into comprehensive, actionable UX reports for developers.

Summary updated

BibTeX
@misc{li2018quantitative,
  author = {Xiaosong Li and Ye Liu and Zizhou Fan and Will Li},
  title = {{A Quantitative Approach in Heuristic Evaluation of E-commerce Websites}},
  year = {2018},
  eprint = {1801.04829},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.04829},
}
Figure