An Analysis of Knowledge Collected from Volunteer Contributors

Timothy Chklovski, Yolanda GilAAAI 2005

Introduction

Building comprehensive common-sense knowledge bases is a fundamental challenge in artificial intelligence. While expert engineering and automated text mining have been the primary methods, this paper explores a third paradigm: mass collaboration via volunteer contributors. Using data from the LEARNER2 system, the authors analyze the trade-offs between data quality and the breadth of coverage when relying on non-expert volunteers.

The Challenge of Broad-Coverage Knowledge

Traditional approaches to knowledge acquisition face significant hurdles. Expert-led projects like WordNet or CYC produce high-quality data but are prohibitively slow and expensive to scale. Conversely, text mining algorithms can process vast amounts of data quickly but often struggle with "common sense"—the basic facts that humans rarely bother to write down because they are so obvious. Volunteer contribution offers a middle ground, but it introduces concerns regarding the reliability of untrained contributors and the efficiency of their efforts.

Methodology: The LEARNER2 System

The researchers deployed LEARNER2 as an interactive kiosk in a science museum. The system prompted visitors with simple templates, such as: "A [car] has a piece or a part called a(n) [____]."

To evaluate the resulting data, the authors analyzed a subset of 6,658 entries representing 2,088 distinct "part-of" statements for 326 everyday objects. These statements were then evaluated by human judges. Two primary metrics were used:

  1. Redundancy: The number of times different contributors provided the exact same statement.
  2. Generation Frequency (gf): The relative frequency of a specific part being mentioned for a given object.

Results: The Power of Redundancy

A key finding of the study is the strong correlation between redundancy and quality. As the number of independent contributors for a specific fact increases, the likelihood of that fact being judged as correct rises significantly.

Table 1: Quality of statements as a function of redundancy.

As shown in the table above, statements contributed only once had a 60% accuracy rate. However, for statements with a redundancy of two or more, accuracy jumped to 89.8%. Facts contributed five or more times reached a near-perfect 97.5% agreement among judges. This suggests that redundancy is a highly effective proxy for truth in crowdsourced environments.

Results: The Saturation of Coverage

While redundancy ensures quality, it also leads to significant inefficiency. The study found that contributors suffer from an "obviousness bias," repeatedly entering the same common facts (e.g., "a hammer has a handle") while neglecting rarer but equally valid information.

Figure 1: The growth of distinct statements relative to total entries.

The growth curve of distinct statements illustrates this diminishing return. As the total number of entries increases, the rate at which new, distinct knowledge is discovered slows down. Nearly 69% of the total effort in the analyzed dataset was spent on redundant entries. Furthermore, the overlap with formal ontologies like WordNet was only 10-15%, indicating that volunteers provide a different type of "typical" knowledge that formal systems often miss.

Conclusion and Future Directions

The analysis of LEARNER2 demonstrates that volunteer contributors can build high-quality knowledge bases if redundancy is leveraged as a quality control mechanism. However, to achieve broad coverage, future systems must implement active guidance. By using "taboo lists" or targeted prompts, systems can steer volunteers away from saturated, obvious facts and toward the "long tail" of rare knowledge. Additionally, addressing the inherent subjectivity of common-sense facts—where judges may disagree on "fuzzy" relationships—remains a critical area for future research.

Reference: Chklovski, T., & Gil, Y. (2005). An Analysis of Knowledge Collected from Volunteer Contributors. Proceedings of the 20th National Conference on Artificial Intelligence (AAAI-05).

Summary updated

BibTeX
@inproceedings{chklovski2005analysis,
  author = {Timothy Chklovski and Yolanda Gil},
  title = {{An Analysis of Knowledge Collected from Volunteer Contributors}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2005},
}
Figure