Measuring the Importance of User-Generated Content to Search Engines

Nicholas Vincent, Isaac Johnson, Patrick Sheehan, Brent HechtarXiv 2019 · arXiv

TL;DR

Google Search is profoundly dependent on volunteer-created content, with Wikipedia serving as the single most prevalent source of information across diverse query types. By auditing search results, researchers found that Wikipedia appears in over 80% of results for trending and controversial topics, highlighting a significant economic asymmetry where multi-billion dollar platforms rely heavily on unpaid "data labor."

What Problem Does This Paper Address?

While we know search engines are immensely profitable, we often overlook the "raw materials" that power them. Recent studies suggested that users have a strong preference for Wikipedia, but a critical gap remained: How often does Google actually surface this content in the first place?

The authors tackle this by performing a rigorous audit of Google Search to quantify its reliance on User-Generated Content (UGC). This research feeds into a larger, high-stakes debate regarding the "technological dividend"—the idea that if intelligent technologies are built on the back of free human labor (like Wikipedia editing), those contributors should perhaps receive a larger share of the resulting economic benefits.

Methods

The researchers developed a custom software framework based on the Selenium-based SerpScrap library to automate desktop Chrome searches. To ensure their results weren't just a byproduct of geographic bubbles, they used two distinct experimental setups:

  1. Geographic Variation Experiment: They simulated queries from specific U.S. counties across three spectra: urban vs. rural, high vs. low income, and political leaning (based on 2016 election results). They achieved this by injecting Javascript to override the browser's geolocation API.
  2. Population-Weighted Experiment: They sampled 40 U.S. counties using a population-weighted approach to provide a reliable "average" view of search results.

Crucially, the team applied two different definitions of UGC:

  • Platform-centric: Anything on sites like Wikipedia, Twitter, or YouTube.
  • Content-centric: A stricter definition (based on OECD standards) requiring that content be created "outside professional routines." They used qualitative coding to determine if a result was truly "amateur" UGC or just a professional/corporate account using a social platform.

Datasets

The study analyzed six categories of high-value queries, selected to represent different dimensions of importance:

  • Popularity: "Trending" queries (from Google Trends) and "Most-Popular" queries (high-volume commercial/political topics).
  • Revenue: High-cost-per-click categories like "Insurance" and "Loans."
  • Influence: "Medical" topics (sampled from Bing’s top health queries) and "Controversial" topics (from ProCon.org).

Key Insights and Ideas

  • Wikipedia is the Backbone of Search: Wikipedia isn't just one of many sources; it is the most prevalent individual domain in the study. For trending and controversial queries, it is nearly ubiquitous.
  • The "UGC" Label is Often a Misnomer: One of the most surprising findings was that 95% of social media results (excluding Wikipedia) failed the "content-centric" definition of UGC. Most tweets or Facebook posts surfaced by Google come from corporate, journalistic, or official political accounts.
  • Wikipedia is the Exception: Unlike other social platforms, Wikipedia remains a bastion of true amateur, volunteer-driven content that provides immense value to search engines.
  • Personalization has Limits: While geographic personalization is known to affect local results (like maps), the authors found it had surprisingly little impact on the prevalence of Wikipedia and other UGC.

Key Results

The data confirms Google's heavy reliance on Wikipedia across all query types. As shown in the summary metrics, Wikipedia's presence is dominant:

This figure summarizes key metrics for all UGC domains and the top 5 non-UGC domains. Wikipedia (top row) shows the highest average full-page incidence rate.
  • Incidence Rates: Wikipedia appeared in 58% of all SERPs (Search Engine Result Pages) on average. For trending and controversial queries, this jumped to over 80%.
  • Top-Three Prominence: In the highly valuable "top three" links, Wikipedia appeared over 50% of the time for trending and controversial categories.
  • The Knowledge Panel Effect: When including links found in the "Knowledge Panel" (the information box on the right side of the screen), Wikipedia’s average full-page incidence rate rose from 58% to 69%.
  • Comparison to Others: Twitter was the next most frequent "UGC platform" (30% incidence), but as noted, most of this was professional content rather than true user-generated data.

Limitations and Future Work

The authors acknowledge several constraints:

  • Geographic Scope: The study was limited to the United States and English-language queries. Wikipedia’s importance likely varies across different languages and national contexts.
  • Device Type: The audit focused exclusively on desktop search. Given the rise of mobile search and its emphasis on local results, mobile SERPs might show different UGC patterns.
  • Query Sample Size: While diverse, the query sets were relatively small (10-20 per category) to avoid triggering Google's rate limits.
  • Longitudinal Data: Search algorithms are constantly evolving (e.g., the introduction of deep learning models like RankBrain). The authors suggest that longitudinal auditing is necessary to see if Google's reliance on Wikipedia is increasing or decreasing over time.

Summary updated

Abstract

Search engines are some of the most popular and profitable intelligent technologies in existence. Recent research, however, has suggested that search engines may be surprisingly dependent on user-created content like Wikipedia articles to address user information needs. In this paper, we perform a rigorous audit of the extent to which Google leverages Wikipedia and other user-generated content to respond to queries. Analyzing results for six types of important queries (e.g. most popular, trending, expensive advertising), we observe that Wikipedia appears in over 80% of results pages for some query types and is by far the most prevalent individual content source across all query types. More generally, our results provide empirical information to inform a nascent but rapidly-growing debate surrounding a highly-consequential question: Do users provide enough value to intelligent technologies that they should receive more of the economic benefits from intelligent technologies?
BibTeX
@misc{vincent2019measuring,
  author = {Nicholas Vincent and Isaac Johnson and Patrick Sheehan and Brent Hecht},
  title = {{Measuring the Importance of User-Generated Content to Search Engines}},
  year = {2019},
  eprint = {1906.08576},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.08576},
}

Loading related content...

Figure