TL;DR
This paper introduces the concept of "collective discourse" – the aggregated opinions and information from many non-expert contributors online – and analyzes its properties. They demonstrate that this collective discourse exhibits diverse perspectives and forms a "small-world" network of interconnected ideas, suggesting it can be a valuable source of knowledge for answering complex questions.
What Problem Does This Paper Address?
The paper tackles the challenge of understanding and leveraging the vast amount of information generated by non-experts on the internet, often referred to as "collective discourse." While individual contributions might be noisy or incomplete, the authors hypothesize that the aggregation of these contributions can reveal emergent properties and collective intelligence. They aim to computationally analyze this phenomenon, specifically focusing on how diverse perspectives within collective discourse contribute to a more comprehensive understanding of a topic, akin to the "wisdom of crowds." This is particularly relevant in an era where social media, review sites, and collaborative platforms are primary sources of information.
Methods
The core of the paper involves constructing and analyzing networks of "factoids" – atomic semantic units representing specific pieces of information or concepts.
- Factoid Extraction: For each dataset, factoids were identified. For movie reviews, these were cast names and plot keywords. For microblogs, news headlines, and citations, human annotators identified key pieces of information.
- Network Construction: Factoids were treated as nodes in a network. An edge was created between two factoids if they co-occurred in a significant number of documents (at least 10).
- Network Analysis: The authors analyzed these factoid networks for "small-world" properties, specifically measuring the clustering coefficient (C) and average shortest path length (l). They compared these metrics to those of random networks with similar parameters.
- Diversity Measurement: The paper quantifies diversity by examining the frequency distribution of factoid mentions across different contributions.
- Application to Question Answering: The authors demonstrate the utility of collective discourse by using it to answer a specific question: determining movie genres. They matched genre names against review mentions and ranked them by frequency.
Datasets
The authors analyzed a range of real-world collective discourse datasets:
- Movie Reviews: User reviews from the Internet Movie Database (IMDB) for 100 randomly selected movies, totaling over 96,500 reviews across 19 genres.
- Microblogs: Tweets related to specific news stories, including tweets about Sarah Palin's divorce rumor and cancellation rumors of TV shows.
- News Headlines: Clusters of headlines from Google News about specific events (e.g., hurricane Bill).
- Citation Sentences: Sentences from academic papers that cite a specific research paper.
Key Insights and Ideas
- Collective Discourse as a Rich Source: The paper argues that aggregating non-expert contributions can reveal a more complete and nuanced understanding of a topic than any single source.
- Diversity is Key: The analysis confirms that collective discourse inherently contains diverse perspectives, a crucial element for "wise crowds" as defined by Surowiecki. Different contributors focus on different aspects of the same subject.
- Factoid Networks Exhibit Small-World Properties: The co-occurrence of factoids forms networks with high clustering and short path lengths. This suggests that related concepts are grouped together, but distant concepts are still easily accessible, mirroring efficient human memory retrieval.
- Non-Experts Can Answer Complex Questions: The experiment on genre prediction demonstrates that even simple aggregation of non-expert opinions (like review mentions) can outperform random guessing and provide surprisingly accurate results for tasks that might otherwise require expert knowledge.
Key Results
- Diversity Confirmed: All analyzed datasets showed a diversity of perspectives, with different factoids being mentioned with varying frequencies.
- Small-World Networks: Factoid co-occurrence networks consistently exhibited higher clustering coefficients and comparable or shorter average path lengths than random networks, confirming small-world properties.
- Genre Prediction Accuracy: Using simple mention frequency from movie reviews, the system achieved significantly better Mean Average Precision (MAP) and F-score for genre extraction compared to random ranking. For example, MAP was 0.698 compared to 0.260 for random ranking. The quality of genre prediction improved rapidly with the number of reviews, showing diminishing returns after about 100 reviews.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the set of factoids extracted for movie reviews (cast names and plot keywords) might not be exhaustive.
- The genre extraction experiment used a simplified approach of matching genre names to review text, which could be further refined.
- Future Work:
- The authors aim to develop models of collective discourse that can capture motivation behind contributions, heterogeneity, and collective intelligence.
- They plan to use computational linguistics, network science, and social theories to predict macro-level discourse phenomena.
- Further research into formulating stochastic models for individual and group behavior is proposed.
Summary updated