TL;DR
Google Search is profoundly dependent on volunteer-created content, with Wikipedia serving as the single most prevalent source of information across diverse query types. By auditing search results, researchers found that Wikipedia appears in over 80% of results for trending and controversial topics, highlighting a significant economic asymmetry where multi-billion dollar platforms rely heavily on unpaid "data labor."
What Problem Does This Paper Address?
While we know search engines are immensely profitable, we often overlook the "raw materials" that power them. Recent studies suggested that users have a strong preference for Wikipedia, but a critical gap remained: How often does Google actually surface this content in the first place?
The authors tackle this by performing a rigorous audit of Google Search to quantify its reliance on User-Generated Content (UGC). This research feeds into a larger, high-stakes debate regarding the "technological dividend"—the idea that if intelligent technologies are built on the back of free human labor (like Wikipedia editing), those contributors should perhaps receive a larger share of the resulting economic benefits.
Methods
The researchers developed a custom software framework based on the Selenium-based SerpScrap library to automate desktop Chrome searches. To ensure their results weren't just a byproduct of geographic bubbles, they used two distinct experimental setups:
- Geographic Variation Experiment: They simulated queries from specific U.S. counties across three spectra: urban vs. rural, high vs. low income, and political leaning (based on 2016 election results). They achieved this by injecting Javascript to override the browser's geolocation API.
- Population-Weighted Experiment: They sampled 40 U.S. counties using a population-weighted approach to provide a reliable "average" view of search results.
Crucially, the team applied two different definitions of UGC:
- Platform-centric: Anything on sites like Wikipedia, Twitter, or YouTube.
- Content-centric: A stricter definition (based on OECD standards) requiring that content be created "outside professional routines." They used qualitative coding to determine if a result was truly "amateur" UGC or just a professional/corporate account using a social platform.
Datasets
The study analyzed six categories of high-value queries, selected to represent different dimensions of importance:
- Popularity: "Trending" queries (from Google Trends) and "Most-Popular" queries (high-volume commercial/political topics).
- Revenue: High-cost-per-click categories like "Insurance" and "Loans."
- Influence: "Medical" topics (sampled from Bing’s top health queries) and "Controversial" topics (from ProCon.org).
Key Insights and Ideas
- Wikipedia is the Backbone of Search: Wikipedia isn't just one of many sources; it is the most prevalent individual domain in the study. For trending and controversial queries, it is nearly ubiquitous.
- The "UGC" Label is Often a Misnomer: One of the most surprising findings was that 95% of social media results (excluding Wikipedia) failed the "content-centric" definition of UGC. Most tweets or Facebook posts surfaced by Google come from corporate, journalistic, or official political accounts.
- Wikipedia is the Exception: Unlike other social platforms, Wikipedia remains a bastion of true amateur, volunteer-driven content that provides immense value to search engines.
- Personalization has Limits: While geographic personalization is known to affect local results (like maps), the authors found it had surprisingly little impact on the prevalence of Wikipedia and other UGC.
Key Results
The data confirms Google's heavy reliance on Wikipedia across all query types. As shown in the summary metrics, Wikipedia's presence is dominant:
- Incidence Rates: Wikipedia appeared in 58% of all SERPs (Search Engine Result Pages) on average. For trending and controversial queries, this jumped to over 80%.
- Top-Three Prominence: In the highly valuable "top three" links, Wikipedia appeared over 50% of the time for trending and controversial categories.
- The Knowledge Panel Effect: When including links found in the "Knowledge Panel" (the information box on the right side of the screen), Wikipedia’s average full-page incidence rate rose from 58% to 69%.
- Comparison to Others: Twitter was the next most frequent "UGC platform" (30% incidence), but as noted, most of this was professional content rather than true user-generated data.
Limitations and Future Work
The authors acknowledge several constraints:
- Geographic Scope: The study was limited to the United States and English-language queries. Wikipedia’s importance likely varies across different languages and national contexts.
- Device Type: The audit focused exclusively on desktop search. Given the rise of mobile search and its emphasis on local results, mobile SERPs might show different UGC patterns.
- Query Sample Size: While diverse, the query sets were relatively small (10-20 per category) to avoid triggering Google's rate limits.
- Longitudinal Data: Search algorithms are constantly evolving (e.g., the introduction of deep learning models like RankBrain). The authors suggest that longitudinal auditing is necessary to see if Google's reliance on Wikipedia is increasing or decreasing over time.
Summary updated