How do users find things with PubMed?: towards automatic utility evaluation with user simulations

Jimmy Lin, Mark D. Smucker

TL;DR

This paper explores how users find information in PubMed, a biomedical literature search engine, by simulating user search strategies. They demonstrate that a "related articles" browsing feature can significantly compensate for poor initial search results, suggesting a non-linear relationship between retrieval quality and overall system utility. This work also proposes user simulations as a valuable, formative tool for evaluating interactive information retrieval systems.

What Problem Does This Paper Address?

PubMed, while a crucial resource for clinicians and scientists, is known for its challenging Boolean retrieval engine and the difficulty users face in formulating effective queries. This often leads to poor initial search results, with a significant percentage of queries returning zero results. The paper aims to understand how users navigate this system and, more broadly, how to evaluate the utility of interactive information retrieval (IR) systems, which goes beyond just the accuracy of a single ranked list. Traditional evaluation methods like Cranfield-style batch evaluations are limited in capturing the dynamic, interactive nature of user search, while full user studies are costly and time-consuming. The authors seek a middle ground: a method to assess system utility that is both rigorous and efficient.

Methods

The core of this research involves user simulations. Instead of real users, the authors model idealized users employing specific search strategies. They developed a two-by-two experimental design, combining two content-similarity algorithms (one based on language modeling, the other mimicking PubMed's own "related articles" feature) with two browsing strategies:

  • Greedy Strategy: Users examine documents sequentially and immediately explore related articles when a relevant document is found. They stop after encountering two non-relevant documents in a row.
  • Breadth-Like Strategy: Users examine documents sequentially but only explore related articles when the current list's quality drops significantly (defined by precision or consecutive non-relevant documents). They also stop after two non-relevant documents.

The simulations generate sequences of documents that a simulated user would examine. The utility of the system is then measured by evaluating the quality of these simulated document sequences using standard IR metrics like interpolated precision at recall 0.5 (IPR50) and P20.

Datasets

The evaluation was conducted using data from the TREC 2005 genomics track. This dataset consists of a ten-year subset of MEDLINE (1994-2003), containing approximately 4.6 million records. The topics were based on generic topic templates (GTTs) designed to represent common information needs in the biomedical domain, with relevance judgments provided by biology students and researchers.

Key Insights and Ideas

  • Browsing as a Compensator: The "related articles" feature is a powerful tool that can significantly mitigate the negative impact of poor initial search results. This suggests that system utility is not solely determined by the initial ranked list but also by the interactive features that allow users to explore and discover relevant information.
  • Non-linear Utility: The relationship between the quality of initial results and the overall utility of the system is not linear. At very low initial result quality, small improvements can lead to large gains in utility (an "amplification effect"). However, as initial results improve, the marginal gain in utility diminishes ("diminishing returns effect").
  • User Simulations as a Formative Tool: User simulations offer a practical and scalable alternative to traditional user studies for formative evaluation. They allow researchers to rapidly explore hypotheses, inform experimental design, and gain insights that might be missed in batch evaluations or be too costly to obtain through extensive user testing.
  • Content Similarity is Key: The specific algorithm used for content similarity (Lemur vs. PubMed's) seemed less critical than the presence of such a browsing mechanism. This suggests that the underlying principle of content-based exploration is robust.
  • Real-World Relevance: Analysis of PubMed query logs and manual query formulation experiments in the paper provide empirical evidence that users do indeed utilize the "related articles" feature and that it plays a significant role in their search behavior.

Key Results

  • Simulations showed substantial improvements in utility (measured by IPR50) when initial results had low precision, particularly with the greedy browsing strategy. For instance, starting with a ranked list with 15% precision, users could gain about eight absolute percentage points in IPR50 by browsing related articles.
  • The study found that for 65% of the trials, the initial P20 score was 0.35 or lower, highlighting the common occurrence of challenging topics or poor initial results.
  • In cases where initial results were poor (P20 below 0.25), browsing related articles rarely hurt user utility (less than 5% of the time).
  • Figure 5 shows that for baseline P20 scores below 0.25, the vast majority of trials resulted in an increase in utility, with only a small fraction showing a decrease.

Limitations and Future Work

  • Simulation Realism: While user simulations offer advantages, their validity hinges on the realism of the simulated user models. The authors acknowledge that their simulations do not capture all aspects of human behavior, such as "lightweight lookahead" behavior, which might prevent users from exploring related articles when initial results are already good.
  • Generalizability: The authors are cautious about drawing broad generalizations from their specific set of manual PubMed queries, acknowledging the wide variation in user information-seeking behavior.
  • Future Work: The paper suggests that user simulations can be used prescriptively, not just descriptively, to educate users on effective search strategies. They also advocate for the development of standard test suites for simulation-based evaluations to ensure comparability across studies, similar to how mean average precision became a standard metric. The authors also hint at the need to explore more sophisticated user models that better capture human search behavior.

Summary updated

Abstract

In the context of document retrieval in the biomedical domain, this paper explores the complex relationship between the quality of initial query results and the overall utility of an interactive retrieval system. We demonstrate that a content-similarity browsing tool can compensate for poor retrieval results, and that the relationship between retrieval performance and overall utility is non-linear. Arguments are advanced with user simulations, which characterize the relevance of documents that a user might encounter with different browsing strategies. With broader implications to IR, this work provides a case study of how user simulations can be exploited as a formative tool for automatic utility evaluation. Simulation-based studies provide researchers with an additional evaluation tool to complement interactive and Cranfield-style experiments.
BibTeX
@inproceedings{lin2008how,
  author = {Jimmy Lin and Mark D. Smucker},
  title = {{How do users find things with PubMed?: towards automatic utility evaluation with user simulations}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2008},
  doi = {10.1145/1390334.1390340},
}

Loading related content...

Figure