LLM-Based Web Data Collection for Research Dataset Creation

Thomas Berkane, Marie-Laure Charpignon, Maimuna S. MajumderEMNLP 2025

TL;DR

This paper introduces an LLM-powered framework for automating the creation of research datasets from web data. It addresses the time-consuming and error-prone nature of manual data collection by integrating LLMs for query generation, web page retrieval, data extraction, and quality control, while keeping users in the loop for oversight.

What Problem Does This Paper Address?

Researchers across various disciplines heavily rely on web data for insights and validation. However, assembling comprehensive datasets from the vast and scattered online information is a significant challenge. Traditional methods involve manual web searches and data extraction, which are not only time-consuming but also prone to human error. Furthermore, recent restrictions on data access from major social media platforms have amplified the need for independent data collection methods. This paper tackles the problem of efficiently and reliably creating structured, research-ready datasets from the web at scale, overcoming the limitations of manual efforts and existing automated tools.

Methods

The core of the framework is a human-in-the-loop system that leverages Large Language Models (LLMs) for automated data collection. The process involves several key stages:

  • Dataset Description & Schema Generation: Users provide a natural language description of their target dataset and a CSV file with placeholder names. An LLM (Anthropic's Claude 3.7 Sonnet) is used to generate a Pydantic schema that defines the structure of the extracted data.
  • Search Query Generation: The framework uses a template-based system. Users define query templates with placeholders, and the LLM generates Google Search queries to find relevant web pages.
  • Web Page Retrieval: The generated queries are executed via Google Search, and search results (titles and URLs) are scraped using Selenium. A re-ranking step is employed to filter pages based on relevance to the original query, mitigating search engine bias. Snowball sampling is used to follow hyperlinks and discover more relevant pages, especially PDF documents.
  • Data Extraction: The framework extracts structured information from retrieved web pages and PDFs (converted to markdown) using an LLM. It incorporates "source grounding" by extracting a short text snippet where the data point was found, enabling user verification and mitigating LLM hallucinations.
  • Quality Control: An LLM performs a sanity check on the extracted data, flagging potential issues for user review. This step helps identify outliers, duplicates, and inconsistencies.
  • Bias Mitigation: The framework includes mechanisms to address recency bias (using time-chunking for date ranges) and geographical bias (allowing users to specify a country for localized search results).

Datasets

The paper evaluates its framework on three diverse data collection tasks:

  1. COVID-19 App Downloads: Collecting reports of download counts for COVID-19 contact tracing apps in different U.S. states from 2020 to 2022.
  2. Climate Events: Gathering data on climate events (earthquakes, floods, droughts) in Haiti and Cameroon from 2021 to 2023.
  3. Police Misconduct: Compiling information on police misconduct cases in Tennessee from 2015 to 2024.

For each task, a reference dataset was constructed by combining manually validated data points from multiple methods (including manual web search) to serve as a benchmark. The paper does not introduce new datasets but rather uses existing data collection needs as benchmarks.

Key Insights and Ideas

  • LLMs as End-to-End Data Collection Agents: The paper demonstrates that LLMs can be effectively integrated into a pipeline that automates the entire process of web data collection, from query formulation to structured dataset creation.
  • Human-in-the-Loop is Crucial: While LLMs automate much of the process, user oversight remains essential for ensuring data quality, aligning with research objectives, and addressing nuanced issues that automated systems might miss.
  • Mitigating LLM Hallucinations: Source grounding, where extracted data is linked to its original text snippet, is a key technique to combat LLM hallucinations and improve data reliability.
  • Addressing Search Engine Biases: The framework actively tackles recency and geographical biases inherent in search engines, which is critical for creating representative datasets.
  • Template-Based Querying for Scalability: The use of query templates with placeholders allows for efficient generation of a large number of structurally similar queries, enabling longitudinal and cross-sectional data collection.

Key Results

  • The LLM-based framework significantly outperforms baselines (ChatGPT Search, ChatGPT Deep Research, and Perplexity Deep Research) in terms of recall and F1-score across all three evaluation tasks. For example, it achieved an F1-score of 82.8% for COVID-19 App Downloads, compared to 45.2% for ChatGPT Deep Research.
  • The quality control step consistently increased the F1-score across all tasks.
  • Recency bias mitigation was particularly effective for the police misconduct task, which spanned a long time period.
  • A user evaluation showed that the framework is useful and intuitive, significantly reducing the time and effort required for data collection compared to manual Google searches.

Limitations and Future Work

  • Data Access Limitations: The framework cannot capture content from sites with access restrictions (e.g., paywalls) and is susceptible to web pages being deleted or modified over time. Future work includes integrating with the Wayback Machine to retrieve historical versions of web pages.
  • Incomplete Coverage: Despite efforts, the framework may still miss some relevant data points due to search engine limitations or filtering. Supplementing with manual searches is recommended for critical applications.
  • Reproducibility Challenges: Reliance on Google Search introduces inherent personalization biases that impact reproducibility.
  • Parameter Tuning: Users may find it difficult to choose appropriate parameters (like the re-ranking threshold) without prior experience.
  • Manual Verification: While automated quality control helps, manual verification of many entries is still necessary.
  • Multimedia Content: The current framework is limited to text and PDF documents. Future work aims to extend support for image and video content.
  • Automated Evaluation: Automating the matching of collected data against reference datasets would improve the efficiency of the evaluation process.
  • Enhanced LLM Capabilities: Future work could involve integrating more powerful LLMs for improved quality control and exploring LLMs that can interact with web interfaces directly.

Summary updated

BibTeX
@inproceedings{berkane2025llm,
  author = {Thomas Berkane and Marie-Laure Charpignon and Maimuna S. Majumder},
  title = {{LLM-Based Web Data Collection for Research Dataset Creation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure