TL;DR
This paper tackles the challenge of ambiguity in social tagging systems (folksonomies) by proposing a method to "semantify" tags. They introduce a model where tags are linked to formal concepts, enabling more precise search and analysis, and demonstrate its effectiveness on a del.icio.us dataset.
What Problem Does This Paper Address?
Social annotation systems like del.icio.us and Flickr, often called folksonomies, allow users to tag resources (e.g., web pages) with keywords. While simple and popular, these systems suffer from significant semantic issues. Tags are free-form text, leading to problems like:
- Ambiguity (Homography): A single tag can have multiple meanings (e.g., "Java" for the island, programming language, or coffee).
- Synonymy: Different tags can mean the same thing (e.g., "image" and "picture").
- Base Form Variation: Different forms of the same word are treated as distinct tags (e.g., "run," "running," "runs").
- Specificity Gap: A search for a general term might miss resources tagged with a more specific, related term (e.g., searching "cheese" might miss "cheddar").
These issues hinder effective search, knowledge organization, and computational analysis of the vast amount of information in folksonomies.
Methods
The authors propose a "semantic folksonomy model" that extends the traditional tripartite user-resource-tag model. The core idea is to move from simple tags to "controlled tags" that link a natural language tag to formal linguistic concepts. This is achieved through a multi-step process:
- Tag Splitting: Decomposing multi-word tags (e.g., "javaisland" into "java" and "island") using WordNet.
- Lemmatization: Reducing words to their base form (e.g., "banks" to "bank").
- Word Sense Disambiguation (WSD): Automatically suggesting the correct sense of a tag token from a controlled vocabulary (WordNet) based on contextual clues from other tags on the same resource and by the same user.
- Manual Validation: Human annotators review and correct the automatically generated splits and disambiguations to create a "golden standard" dataset.
They then use this semantically annotated dataset to evaluate the impact of formal semantics on search quality.
Datasets
- del.icio.us dataset: A large crawl of del.icio.us bookmarks was used.
- Golden Standard Dataset: A subset of the del.icio.us data was manually annotated by the authors to create a high-quality, semantically disambiguated dataset for evaluation. This dataset consists of 4,707 tag annotations with 871 unique tags on 299 URLs from 172 different web domains, selected based on specific criteria to reduce spam and ensure sufficient user overlap.
- WordNet: Used as the underlying controlled vocabulary for disambiguation.
Key Insights and Ideas
- The "Semantification" of Tags: The core idea is that by linking free-form tags to formal concepts, we can overcome the inherent ambiguities of natural language in social tagging systems.
- Context is King for Disambiguation: The WSD algorithm leverages multiple levels of context (tokens within the same tag, other tags on the resource, and tags from other users on the same resource) to improve disambiguation accuracy.
- Controlled Vocabularies Have Limitations: While WordNet is a valuable resource, it only covers about 50% of the terms in their folksonomy dataset, highlighting the dynamic nature of user vocabularies and the inadequacy of static vocabularies for full coverage.
- Semantic Search Significantly Improves Quality: Linking tags to concepts dramatically improves search precision and recall compared to keyword-based search, addressing issues like homography and synonymy.
- Manual Effort is Still Crucial: While automatic methods are helpful, manual validation remains essential for creating high-quality semantically annotated datasets, especially given the limitations of current WSD accuracy.
Key Results
- Preprocessing Accuracy: The preprocessing step (tag splitting and lemmatization) achieved 80.31% accuracy.
- WSD Accuracy: The proposed WSD algorithm achieved 59.37% accuracy. Accuracy was higher (around 90%) for tokens with fewer possible senses and dropped below 50% for tokens with many senses.
- Annotator Agreement: Inter-coder agreement for disambiguation was 0.76, indicating a reasonable level of consistency among human annotators.
- Controlled Vocabulary Coverage: Less than half (48.7%) of the validated terms could be mapped to a sense in WordNet.
- Search Improvement: Concept-based search queries demonstrated significantly improved precision and recall compared to keyword-based queries, especially as the number of query terms increased. For example, with one query term, keyword search precision was around 76%, while concept-based search, by definition, yields correct and complete results.
Limitations and Future Work
- Manual Annotation Effort: The paper acknowledges that a fully automatic conversion of existing folksonomies to the formal model is unlikely; significant manual effort is required.
- Static Vocabularies: The reliance on WordNet, a static vocabulary, limits coverage and suggests the need for more dynamically evolving vocabularies.
- Dataset Size and Specificity Gap: The current dataset, while useful, is not large enough to fully demonstrate the impact of the "specificity gap" in search. The authors plan to extend their study to other domains (cooking, education, travel) to observe this effect.
- User Motivation: The paper notes that users do not provide a large amount of annotations, and further research is needed to understand why and to motivate users to provide more diverse and semantically rich annotations.
- WSD Accuracy: The WSD accuracy, while improved by context, is still not perfect, suggesting that users may need to manually check or provide input during the annotation phase.
Summary updated