Geovicla: Automated Classification of Interactive Web-Based Geovisualizations

Phil Hüffer, Auriol Degbelo, Benjamin RisseGIScience 2025 · DOI

TL;DR

The paper introduces Geovicla, a new dataset of over 2,500 web pages classified as interactive visualizations, interactive geovisualizations, or neither. They then compare three automated classification methods, finding that LLM embeddings combined with a multilayer perceptron achieve the best performance, highlighting the potential of advanced NLP for discovering online geospatial data.

What Problem Does This Paper Address?

The internet is awash with interactive visualizations, including maps and geovisualizations, which are invaluable for exploring and understanding geospatial information. However, finding these resources is a significant challenge. Existing search engines and cataloging systems often struggle to effectively identify and categorize these dynamic web content types. This lack of discoverability hinders researchers and users from synthesizing information, generating hypotheses, and accessing crucial geospatial data. The paper aims to address this by developing and evaluating automated methods for classifying webpages that contain interactive geovisualizations, making them more accessible.

Methods

The authors explored three distinct approaches for classifying webpages:

  • Heuristic-based approach: This method relies on manually crafted rules and heuristics, likely examining the HTML code and content for specific patterns or keywords indicative of interactive visualizations.
  • Feature-engineering approach: This involves creating hand-crafted feature vectors from webpage elements (like text content, external links, and HTML tags) and then feeding these into a machine learning classifier. They specifically looked at keywords related to visualization frameworks, HTML element IDs/classes, and descriptive sentences.
  • Embedding-based approach: This cutting-edge method leverages Large Language Models (LLMs) to automatically generate dense vector representations (embeddings) of webpage content. These embeddings capture semantic meaning and are then used by machine learning classifiers. They experimented with the "stella 1.5b" LLM, considering its performance and context length.

For classification, they tested several standard machine learning models, including k-Nearest Neighbors (kNN), Support Vector Machines (SVM), Naive Bayes, Random Forest, and Multi-layer Perceptrons (MLP). They evaluated performance using F1 scores (with macro averaging due to dataset imbalance) and ROC-AUC, comparing both multiclass (IGV vs. IV vs. noIV) and binary (IGV vs. noIV) classification strategies.

Datasets

The primary contribution in terms of datasets is the Geovicla dataset. This dataset comprises approximately 2,500 web pages, manually labeled into three categories:

  • Interactive Visualisation (IV): Webpages displaying at least one visualization that allows for computer-mediated interaction.
  • Interactive Geovisualisation (IGV): Webpages showing at least one geovisualization that allows for computer-mediated interaction.
  • No Interactive Visualisation (noIV): Webpages that do not fall into the above categories.

The dataset was constructed by generating search queries using ChatGPT, combining synonyms for "interactive visualization" with various topics (e.g., climate change, sustainable agriculture). These queries were used with the Google Custom Search API to harvest webpages, which were then manually labeled. The dataset is available in CSV and JSON formats, with options to include original HTML code and embeddings.

Key Insights and Ideas

  • LLMs are powerful for semantic understanding: The results strongly suggest that LLM embeddings are significantly better at capturing the nuances of webpage content for classification compared to hand-crafted features. This highlights the growing importance of advanced NLP in understanding complex web content.
  • The "black box" trade-off: While LLM embeddings offer superior performance, they come at the cost of interpretability. The authors acknowledge this trade-off, noting that feature-engineering methods are more transparent and computationally cheaper at runtime, even if their performance is lower.
  • Context length matters for LLMs: The study found that a significant portion of web documents exceeded the context length of the chosen LLM. This necessitates further investigation into how LLMs handle longer texts and the impact of context length on classification accuracy.
  • Interactive geovisualization classification is challenging: Even with advanced methods, achieving very high accuracy remains difficult, indicating that this is a complex problem requiring further research.

Key Results

  • Best performance with LLM embeddings: The embedding-based approach, particularly using LLM embeddings with a Multi-layer Perceptron (MLP), achieved the highest accuracy.
  • For multiclass classification, the best performance reached up to 74% accuracy and 75% F1 score.
  • For binary classification, the best performance reached up to 75% F1 score.
  • Feature engineering limitations: The feature-engineering approach yielded lower performance, with F1 scores generally in the 40%-60% range for multiclass classification.
  • Model performance: For the feature-engineering approach, most models performed similarly. For the embedding-based approach, the MLP generally showed strong performance, often achieving the highest F1 scores.
  • Balancing and context length impact: The effect of balancing the dataset was mixed, sometimes improving and sometimes deteriorating performance. The impact of context length on LLM performance showed small drops, suggesting it's an area for further study.

Limitations and Future Work

Limitations: * Dataset size: While larger than previous efforts, the Geovicla dataset is still considered relatively small compared to standard machine learning benchmarks. * Manual annotation challenges: The manual labeling process was time-consuming, and some webpages were difficult to access or inspect due to loading times, complex hierarchies, or specific display requirements (e.g., large screens). * Dashboard bias: A portion of the dataset came from dashboards, and the potential bias this introduces needs further systematic assessment.

Future Work: * Crowd-sourcing for annotation: To scale up the dataset, the authors suggest exploring crowd-sourcing approaches for data collection and annotation. This could also lead to more fine-grained annotations (e.g., identifying the number and location of visualizations). * Improving harvesting efficiency: The low hit rates observed during the initial harvesting phase indicate a need for more efficient methods. * Exploring other representation methods: Investigating URL-based approaches and combining link and content information could further enhance classification. * Graph-based modeling: The authors propose exploring graph-based modeling of interactive geovisualizations, potentially using graph neural networks, to better capture relationships and improve detection. * Investigating LLM context length impact: Further research is needed to understand and mitigate the effects of document length on LLM performance. * Expanding classifier families: Exploring other classification algorithms and kernel functions could yield further improvements.

Summary updated

BibTeX
@inproceedings{huffer2025geovicla,
  author = {Phil Hüffer and Auriol Degbelo and Benjamin Risse},
  title = {{Geovicla: Automated Classification of Interactive Web-Based Geovisualizations}},
  booktitle = {GIScience},
  year = {2025},
  doi = {10.4230/lipics.giscience.2025.10},
}
Figure