Overview of the INEX 2012 Social Book Search Track

Marijn Koolen, Gabriella Kazai, Jaap Kamps, Michael Preminger, Antoine Doucet, Monica LandoniCLEF (Online Working Notes/Labs/Workshop) 2012

TL;DR

This paper presents the INEX 2012 Social Book Search Track, which evaluated systems for searching and navigating digitized books using both professional and user-generated metadata. The key innovation was exploring how user profiles and the nuances of social book recommendations (like post-catalogued suggestions) influence search relevance, moving beyond simple topical matching.

What Problem Does This Paper Address?

The explosion of digitized book collections (e.g., Google Books) has created a need for effective ways to search and interact with this vast content. Traditional book search relies on formal metadata, but users often seek books based on more subjective criteria like "engaging," "fun," or "well-written." Social media and user-generated content (reviews, tags, forum discussions) offer rich, albeit noisy, signals for these nuanced information needs. This paper addresses the challenge of building search systems that can effectively leverage this diverse metadata, including user preferences and social interactions, to improve book discovery.

Methods

The INEX 2012 Social Book Search Track focused on two main tasks:

  • Social Book Search (SBS): This task investigated complex information needs that go beyond topical relevance, incorporating aspects like genre, recency, engagement, and user profiles. It used a corpus derived from Amazon and LibraryThing, enriched with professional metadata. The core of the SBS task involved analyzing LibraryThing forum discussions where users request book recommendations.
  • Prove It (PI): This task focused on evaluating focused retrieval approaches for finding specific book sections that confirm or refute a factual claim.

For the SBS task, a novel aspect was the use of "touchstone suggestions" from LibraryThing forums as relevance judgments. The researchers also explored differentiating relevance based on whether the topic creator themselves catalogued a suggested book (post-catalogued suggestions, PCS) versus suggestions from other users. User profiles, containing catalogued books and tags, were provided to participants to encourage systems that incorporate user behavior.

Datasets

  • Amazon/LibraryThing Corpus: A large collection of 2.8 million book records from Amazon, enriched with social metadata (tags, ratings, reviews) from LibraryThing. This corpus was extended with professional metadata from the Library of Congress and the British Library.
  • LibraryThing Forum Data: A selection of forum threads where users requested book recommendations. These threads served as the basis for the SBS topics.
  • PCS Topics: A subset of the LibraryThing forum topics specifically chosen for their post-catalogued suggestions, used for evaluation.

Key Insights and Ideas

  • User-Generated Content is Crucial, But Complex: While social metadata (tags, reviews) is valuable, its subjective and sometimes noisy nature makes it challenging to integrate effectively.
  • Beyond Topical Relevance: Users' book search needs are multifaceted, encompassing personal preferences, affective qualities, and social signals, which traditional keyword matching struggles to capture.
  • Post-Catalogued Suggestions as Strong Relevance Signals: When a user adds a suggested book to their personal catalog after starting a recommendation thread, it's a strong indicator of relevance, suggesting a deeper engagement than mere topical agreement.
  • User Profiles Don't Always Improve Performance (Yet): Despite providing user profiles, the best-performing systems in the SBS task were still plain text retrieval systems, indicating that effectively integrating user information remains an open challenge.
  • Genre Influences Suggestions: Fiction topics tend to generate more suggestions than non-fiction topics, and post-cataloguing is more prevalent in fiction-related threads.

Key Results

  • SBS Task: The best performing run used all topic fields (title, narrative, etc.) and document fields, demonstrating the continued importance of comprehensive indexing. However, systems that incorporated user profile information did not consistently outperform baselines.
  • PI Task: The paper notes that the PI task had a low number of participants, limiting the scope of evaluation.
  • Relevance Judgements: The study found that while differentiating relevance based on post-catalogued suggestions (PCS) created a slightly different topic set, it did not significantly alter system rankings compared to using a flat relevance scheme. However, the PCS topics themselves were found to be somewhat different in nature.

Limitations and Future Work

  • User Profile Integration: The paper acknowledges that current systems have not yet effectively leveraged user profile information to improve retrieval performance over strong baselines. Future work will continue to investigate this, potentially using Mechanical Turk for richer relevance judgments and exploring more sophisticated ways to incorporate user behavior.
  • Prove It Task Participation: The low number of participants in the PI task limits its current impact and collaborative potential. Future efforts aim to increase participation and vitalize the task.
  • Relevance Judgment Development: The researchers plan to further develop relevance judgment methods, including aspectization of topics and using crowdsourcing (Mechanical Turk) to gather more detailed and consistent assessments.

Summary updated

BibTeX
@inproceedings{koolen2012overview,
  author = {Marijn Koolen and Gabriella Kazai and Jaap Kamps and Michael Preminger and Antoine Doucet and Monica Landoni},
  title = {{Overview of the INEX 2012 Social Book Search Track}},
  booktitle = {CLEF (Online Working Notes/Labs/Workshop)},
  year = {2012},
}
Figure