A Hierarchical Entity-Based Approach to Structuralize User Generated Content in Social Media: A Case of Yahoo! Answers

Baichuan Li, Jing Liu, Chin-Yew Lin, Irwin King, Michael R. Lyu

Introduction

Social media platforms such as Yahoo! Answers and Reddit have become massive repositories of human knowledge. However, the sheer volume of User Generated Content (UGC) presents a significant organization problem. Most platforms present information in flat lists or broad, static categories, forcing users to manually read, categorize, and synthesize hundreds of results to answer a complex query. Traditional automated solutions like Latent Dirichlet Allocation (LDA) topic models often produce clusters that are mathematically sound but difficult for humans to interpret, while social tagging remains too sparse and inconsistent for reliable navigation.

This research proposes a novel framework to transform unstructured UGC into a Cluster Entity Tree (CET). By leveraging large-scale entity repositories, the approach creates a browsable hierarchy that is both machine-readable and intuitively navigable for human users.

Methodology: Constructing the Cluster Entity Tree

The construction of a CET involves a three-step process designed to move from raw text to a structured hierarchy:

  1. Entity Extraction: Rather than relying on standard Named Entity Recognition (NER) tools that struggle with the informal nature of social media, the authors utilize a large-scale repository (Freebase). They employ a "Stanford Parser" to identify noun phrases and apply an entropy-based filter. This filter calculates the distribution of an entity across categories; high-entropy terms (like "we") are discarded as noise, while low-entropy terms (like "shawarma") are retained as meaningful entities.
  2. Tree Construction: The hierarchy is built based on entity co-occurrence. Starting with a root entity (e.g., "Edinburgh"), Layer 1 consists of entities that frequently appear in questions alongside the root. Layer 2 contains entities that co-occur with both their parent node and the root, creating a logical drill-down structure.
  3. Hierarchical Entity Clustering: To manage the width of the tree, the system uses agglomerative clustering to group similar entities. For instance, "London," "Niddry Street," and "City Center" might be grouped under a "Location" cluster using pattern-based and distributional similarity.
Figure 1: An example of a Cluster Entity Tree (CET) for the root entity 'Edinburgh', showing hierarchical layers and clustered entities.

Experimental Setup and Datasets

The framework was tested using a massive crawl of Yahoo! Answers containing 54.7 million questions across 26 top-level categories. To evaluate performance, the researchers developed two specific subsets: Set EE (520 manually labeled questions for extraction testing) and Set EC (entities mapped to Freebase categories to validate clustering accuracy).

Key Results

Extraction Accuracy

The proposed entity extraction method significantly outperformed baseline models. While standard NER tools achieved an F1 score of only 0.257 on informal social media text, the authors' method reached an F1 score of 0.719, largely due to superior recall (0.809 vs. 0.155).

Figure 3: Comparison of entity extraction performance (Precision, Recall, and F1) between the proposed method and standard baselines.

User and System Performance

The impact of the CET was measured across two dimensions:

  • User Knowledge Acquisition: In tasks where users were asked to learn about a topic, those using the CET-based interface achieved a precision of 0.38, exactly double the 0.19 achieved by users with a traditional list-based search interface.
  • Search Engine Enhancement: By using the CET structure to re-rank search results, the researchers improved the Mean Reciprocal Rank (MRR) of standard Information Retrieval models by 8.2% to 9.3%, demonstrating that the hierarchical entity relationships help surface more relevant content.

Conclusion

The CET framework demonstrates that real-world entities are more effective for organizing UGC than latent mathematical topics. By providing a structured, browsable hierarchy, the system reduces cognitive load for users and improves the performance of automated search engines. Future work will focus on entity canonicalization (e.g., merging "NYC" and "New York City") and integrating automated summarization to provide knowledge snippets directly within the tree nodes.

Summary updated

Abstract

Social media like forumsand microblogshave accumulated a huge amount of user generated content (UGC) containing human knowledge. Currently, most of UGC is listed as a whole or in pre-defined categories. This “list-based” approach is simple, but hinders users from browsing and learning knowledge of certain topics effectively. To address this problem,we propose a hierarchical entity-based approach for structuralizing UGC in social media. By usinga large-scaleentityrepository,wedesign a three-step framework to organize UGC in a novel hierarchical structure called “cluster entity tree (CET)”. With Yahoo! Answers as a test case, we conduct experiments and the results show the effectiveness of our framework in constructingCET. We furtherevaluate the performance of CET on UGC organization in both user and system aspects. From a user aspect, our user study demonstrates that, with CET-based structure, users perform significantlybetter in knowledgelearningthan using traditional list-based approach. From a system aspect, CET substantially boosts the performance of two information retrieval models (i.e., vector space model and query likelihood language model).
BibTeX
@inproceedings{li2013hierarchical,
  author = {Baichuan Li and Jing Liu and Chin-Yew Lin and Irwin King and Michael R. Lyu},
  title = {{A Hierarchical Entity-Based Approach to Structuralize User Generated Content in Social Media: A Case of Yahoo! Answers}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1159},
}
Figure