Generative AI's aggregated knowledge versus web-based curated knowledge

Ted Selker, Yonghui WuarXiv 2024 · DOI · arXiv

TL;DR

This paper investigates the functional boundaries between traditional web search (curated knowledge) and Generative AI (aggregated knowledge). Through a car-buying experiment and a 12-persona taxonomy, the authors find that while GenAI significantly speeds up broad synthesis and presentation tasks, traditional search remains essential for niche facts, real-time data, and verifiable provenance.

What Problem Does This Paper Address?

We are currently witnessing a paradigm shift in how humanity accesses information. For decades, the "Search" paradigm—connecting keywords via Boolean operators to a list of curated sources—was the gold standard. Now, the "Generative" paradigm—where LLMs aggregate and package knowledge into a narrative—is challenging that dominance.

The authors tackle a fundamental question: When should we use which? They aim to move beyond the hype to understand the cognitive trade-offs between these two tools. Specifically, they look at how the "compelling nature" of GenAI responses affects decision-making speed versus the "veracity and provenance" offered by traditional search engines like Google.

Methods

The researchers employed a two-pronged qualitative and quantitative approach:

  1. Consumer Experiment: Five participants were tasked with a complex, real-world decision: buying a car. The experiment was split into three tasks: using only Google Search, using only ChatGPT (v3.5), and using a hybrid of both. Researchers tracked time-on-task, used "think-aloud" protocols, and conducted pre- and post-test surveys to gauge user sentiment and perceived difficulty.
  2. Persona Probing: To generalize their findings, the authors developed a taxonomy of 12 knowledge-seeking personas (e.g., The Fact Checker, The DIY Person, The Hobbyist, The Scientist). They ran hundreds of probe queries across both platforms to determine which paradigm better served the specific goals of each persona.

Datasets

The paper does not use standard machine learning benchmarks. Instead, it generates its own qualitative "dataset" through:

  • Real-world car-buying scenarios: Tailored to different lifestyles (urban luxury vs. rural utility).
  • Persona-based query sets: A custom collection of hundreds of queries designed to test specific knowledge types, from "how to fix a car that won't start" (DIY) to "latest findings in gene therapy" (Professional).

Key Insights and Ideas

  • The Speed-Accuracy Trade-off: GenAI is a "powerhouse" for synthesis. Users familiar with the tool completed tasks significantly faster because the AI "packages" the solution into a single readable narrative, reducing the cognitive load of clicking through multiple links.
  • The "Hallucination" as a Feature and a Bug: The authors note that GenAI uses a "model of persuasion." It fills gaps with plausible-sounding ideas to maintain the flow of conversation. While this makes the interaction feel "natural," it makes the tool less reliable for "Fact Checkers" or "DIYers" who need specific, high-stakes technical instructions.
  • Search for Niche, AI for Broad: Traditional search excels at "very specific, less popularly-known knowledge." For example, when asking how to replace a fender-well on a specific car model, Google provided video demonstrations and exact tool lists, while GenAI gave generic, sometimes inappropriate, steps.
  • The "Backseat Driving" Feeling: The authors describe using GenAI for complex tasks as "trying to drive a car from the back seat." You can give directions, but you lack the direct control over the "raw data" that a search engine provides.

Key Results

  • Statistical Significance in Speed: A t-test comparing Task 1 (Search) and Task 2 (ChatGPT) yielded a p-value of 0.030, indicating a statistically significant difference in time taken. Users were generally faster with ChatGPT, especially those who were already "daily users" or subscribers.
  • User Preference: Almost all participants expressed a preference for the GenAI experience post-study, citing its ability to "brainstorm" and provide "concise answers upon the first request."
  • The Hybrid Advantage: While the data didn't show a significant time difference for the hybrid task (Task 3), qualitative feedback suggested that using both tools provided a "more comprehensive list of options" and helped users verify the AI's suggestions against real-world data.

Limitations and Future Work

The authors acknowledge several limitations:

  • Small Sample Size: The car-buying experiment involved only five participants, which limits the generalizability of the quantitative time-on-task data.
  • Knowledge Cutoff: The study used ChatGPT 3.5, which had a knowledge cutoff of September 2021. Participants frequently noted that the AI's data was "too old" for current car models, a limitation that newer, web-connected models (like GPT-4o or Perplexity) partially address.
  • Future Directions: The authors propose the development of "Knowledge Development Environments"—interfaces that better integrate the provenance of search with the synthesis of GenAI to help users "assemble and evaluate alternative ideas" without sacrificing truthfulness.

Summary updated

Abstract

his paper explores what kinds of questions are best served by the way generative AI (GenAI) using Large Language Models(LLMs) that aggregate and package knowledge, and when traditional curated web-sourced search results serve users better. An experiment compared product searches using ChatGPT, Google search engine, or both helped us understand more about the compelling nature of generated responses. The experiment showed GenAI can speed up some explorations and decisions. We describe how search can deepen the testing of facts, logic, and context. We show where existing and emerging knowledge paradigms can help knowledge exploration in different ways. Experimenting with searches, our probes showed the value for curated web search provides for very specific, less popularly-known knowledge. GenAI excelled at bringing together knowledge for broad, relatively well-known topics. The value of curated and aggregated knowledge for different kinds of knowledge reflected in different user goals. We developed a taxonomy to distinguishing when users are best served by these two approaches.
BibTeX
@misc{selker2024generative,
  author = {Ted Selker and Yonghui Wu},
  title = {{Generative AI's aggregated knowledge versus web-based curated knowledge}},
  year = {2024},
  eprint = {2410.12091},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.12091},
}
Figure