The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language Models

Alexandra Hotti, Riccardo Sven Risuleo, Stefan Magureanu, Aref Moradi, Jens Lagergren

This paper, a collaboration between Klarna and KTH Royal Institute of Technology, introduces a massive new benchmark for web automation and proposes a high-performance hybrid approach combining Graph Neural Networks (GNNs) with Large Language Models (LLMs).

TL;DR

The authors release the Klarna Product Page Dataset, the largest labeled webpage dataset of its kind, featuring over 50,000 pages from 8,000+ merchants. They demonstrate that while simple GNNs are surprisingly effective at identifying web elements, a two-stage pipeline using an LLM to "refine" the GNN’s top candidates boosts nomination accuracy by a staggering 16.8 percentage points.

What Problem Does This Paper Address?

The goal of web automation is to create agents that can navigate the web—filling out forms, closing pop-ups, or finding product details—without human intervention. Central to this is web element nomination: the ability to pick the one specific element (like the "Add to Cart" button) out of thousands of candidates on a page.

The field faces two major hurdles:

  1. Data Scarcity: Most existing datasets are small, simulated, or limited to a handful of websites. Real-world e-commerce pages are messy, featuring thousands of DOM nodes and diverse layouts that vary across regions.
  2. The "Needle in a Haystack" Problem: On a page with 1,000 elements, a classifier with 99% accuracy will still give you 10 false positives. In web automation, you usually only get one shot to click the right button.

Methods

The researchers tackle these problems with a multi-pronged approach:

  • GNN Benchmarking: They adapted and tested several GNN architectures (including GCNs, Tree-LSTMs, and Transformers) to process the Document Object Model (DOM) tree of webpages.
  • LLM Refinement Step: Recognizing that LLMs are too expensive to process an entire webpage's HTML, they proposed a two-stage pipeline. First, a fast GNN filters the page down to the top 10 candidates for a label. Then, GPT-4 examines the local HTML of just those 10 elements to make the final "nomination."
  • Challenge Nomination Training Procedure (CNTP): They introduced a novel training strategy based on "hard-example augmentation." Instead of training on random negative samples, the model is periodically challenged with the unlabeled elements it is most "confused" by (i.e., those it incorrectly ranks highly), forcing it to learn finer distinctions.

Datasets

The core contribution is the Klarna Product Page Dataset:

  • Scale: 51,701 manually labeled product pages from 8,175 distinct e-commerce merchants.
  • Diversity: Covers 8 geographic markets (US, GB, SE, NL, FI, NO, DE, AT) and multiple languages.
  • Labels: Five key elements per page: Price, Name, Main Image, Buy Button, and Cart Button.
  • Formats: Provided as MHTML (raw snapshots), WTL (rendered DOM trees with bounding boxes), and screenshots for computer vision research.

Key Insights and Ideas

  • Simplicity is Substantial: Surprisingly, a simple 2-layer GCN-Mean model outperformed more complex state-of-the-art methods (like FreeDOM) by 9.2 percentage points. This suggests that for web elements, the most relevant context is often concentrated in the immediate local neighborhood of the DOM tree.
  • Classification $\neq$ Nomination: Standard cross-entropy loss doesn't necessarily produce good rankings. The authors' CNTP method bridges this gap by specifically training the model to suppress "confusing" elements that would otherwise rise to the top of the rankings.
  • LLMs as the "Final Judge": LLMs excel at reasoning about HTML semantics but struggle with long contexts. By using a GNN as a "retriever" and the LLM as a "reranker," the authors get the best of both worlds: speed and high-level reasoning.

Key Results

  • GNN Performance: The GCN-Mean model achieved the highest average accuracy among GNNs, reaching 71% without text features and 73% with them.
  • The LLM Boost: Adding the GPT-4 refinement step to the GCN-Mean model increased average nomination accuracy from 64.6% to 81.4% (a 16.8 p.p. jump) on a challenging test subset.
  • Training Gains: The CNTP training procedure boosted the GCN-Mean model's accuracy by approximately 5 percentage points, pushing it past the 80% mark on the validation set.

Limitations and Future Work

  • Single Label Constraint: The dataset currently only provides one ground-truth label per class per page. In reality, multiple elements (e.g., the button text and the button container) might be valid targets. This makes the current evaluation metrics slightly pessimistic.
  • LLM Cost: The LLM refinement was only tested on a subset of 100 pages due to budget constraints. Scaling this efficiently remains a challenge.
  • Vision Integration: While the authors provided screenshots, they primarily focused on HTML/DOM-based models. Future work could explore multi-modal approaches that "see" the page layout just as a human does.

Summary updated

Abstract

Description The Klarna Product Page Dataset is a dataset of publicly available pages corresponding to products sold online on various e-commerce websites. The dataset contains offline snapshots of 51,701 product pages collected from 8,175 distinct merchants across 8 different markets (US, GB, SE, NL, FI, NO, DE, AT) between 2018 and 2019. On each page, analysts labelled 5 elements of interest: the price of the product, its image, its name and the add-to-cart and go-to-cart buttons (if found). These labels are present in the HTML code as an attribute called klarna-ai-label taking one of the values: Price, Name, Main picture, Add to cart and Cart. The snapshots are available in 3 formats: as MHTML files (~24GB), as WebTraversalLibrary (WTL) snapshots (~7.4GB), and as screeshots (~8.9GB). The MHTML format is less lossy, a browser can render these pages though any Javascript on the page is lost. The WTL snapshots are produced by loading the MHTML pages into a chromium-based browser. To keep the WTL dataset compact, the screenshots of the rendered MTHML are provided separately; here we provide the HTML of the rendered DOM tree and additional page and element metadata with rendering information (bounding boxes of elements, font sizes etc.). The folder structure of the screenshot dataset is identical to the one the WTL dataset and can be used to complete the WTL snapshots with image information. For convenience, the datasets are provided with a train/test split in which no merchants in the test set are present in the training set. Corresponding Publication For more information about the contents of the datasets (statistics etc.) please refer to the following TMLR paper. GitHub Repository The code needed to re-run the experiments in the publication accompanying the dataset can be accessed here. Citing If you found this dataset useful in your research, please cite the paper as follows: @article{hotti2024the, title={The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language Models}, author={Alexandra Hotti and Riccardo Sven Risuleo and Stefan Magureanu and Aref Moradi and Jens Lagergren}, journal={Transactions on Machine Learning Research}, issn={2835-8856}, year={2024}, url={https://openreview.net/forum?id=zz6FesdDbB}, note={} }
BibTeX
@article{hotti2024klarna,
  author = {Alexandra Hotti and Riccardo Sven Risuleo and Stefan Magureanu and Aref Moradi and Jens Lagergren},
  title = {{The Klarna Product Page Dataset: Web Element Nomination with Graph Neural Networks and Large Language Models}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=zz6FesdDbB},
}
Figure